cifs: improve read performance for page size 64KB & cache=strict & vers=2.1+

Found a read performance issue when linux kernel page size is 64KB.
If linux kernel page size is 64KB and mount options cache=strict &
vers=2.1+, it does not support cifs_readpages(). Instead, it is using
cifs_readpage() and cifs_read() with maximum read IO size 16KB, which is
much slower than read IO size 1MB when negotiated SMB 2.1+. Since modern
SMB server supported SMB 2.1+ and Max Read Size can reach more than 64KB
(for example 1MB ~ 8MB), this patch check max_read instead of maxBuf to
determine whether server support readpages() and improve read performance
for page size 64KB & cache=strict & vers=2.1+, and for SMB1 it is more
cleaner to initialize server->max_read to server->maxBuf.

The client is a linux box with linux kernel 4.2.8,
page size 64KB (CONFIG_ARM64_64K_PAGES=y),
cpu arm 1.7GHz, and use mount.cifs as smb client.
The server is another linux box with linux kernel 4.2.8,
share a file '10G.img' with size 10GB,
and use samba-4.7.12 as smb server.

The client mount a share from the server with different
cache options: cache=strict and cache=none,
mount -tcifs //<server_ip>/Public /cache_strict -overs=3.0,cache=strict,username=<xxx>,password=<yyy>
mount -tcifs //<server_ip>/Public /cache_none -overs=3.0,cache=none,username=<xxx>,password=<yyy>

The client download a 10GbE file from the server across 1GbE network,
dd if=/cache_strict/10G.img of=/dev/null bs=1M count=10240
dd if=/cache_none/10G.img of=/dev/null bs=1M count=10240

Found that cache=strict (without patch) is slower read throughput and
smaller read IO size than cache=none.
cache=strict (without patch): read throughput 40MB/s, read IO size is 16KB
cache=strict (with patch): read throughput 113MB/s, read IO size is 1MB
cache=none: read throughput 109MB/s, read IO size is 1MB

Looks like if page size is 64KB,
cifs_set_ops() would use cifs_addr_ops_smallbuf instead of cifs_addr_ops,

	/* check if server can support readpages */
	if (cifs_sb_master_tcon(cifs_sb)->ses->server->maxBuf <
			PAGE_SIZE + MAX_CIFS_HDR_SIZE)
		inode->i_data.a_ops = &cifs_addr_ops_smallbuf;
	else
		inode->i_data.a_ops = &cifs_addr_ops;

maxBuf is came from 2 places, SMB2_negotiate() and CIFSSMBNegotiate(),
(SMB2_MAX_BUFFER_SIZE is 64KB)
SMB2_negotiate():
	/* set it to the maximum buffer size value we can send with 1 credit */
	server->maxBuf = min_t(unsigned int, le32_to_cpu(rsp->MaxTransactSize),
			       SMB2_MAX_BUFFER_SIZE);
CIFSSMBNegotiate():
	server->maxBuf = le32_to_cpu(pSMBr->MaxBufferSize);

Page size 64KB and cache=strict lead to read_pages() use cifs_readpage()
instead of cifs_readpages(), and then cifs_read() using maximum read IO
size 16KB, which is much slower than maximum read IO size 1MB.
(CIFSMaxBufSize is 16KB by default)

	/* FIXME: set up handlers for larger reads and/or convert to async */
	rsize = min_t(unsigned int, cifs_sb->rsize, CIFSMaxBufSize);
Reviewed-by: Pavel Shilovsky <pshilov@microsoft.com>
Signed-off-by: Jones Syue <jonessyue@qnap.com>
Signed-off-by: Steve French <stfrench@microsoft.com>

diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c
index 140efc1a937469e73ac801820625ad18ab01705d..182b864b3075bdd1f0b079f9047c8a0f4a5d796d 100644 (file)
--- a/fs/cifs/cifssmb.c
+++ b/fs/cifs/cifssmb.c
@@ -594,6 +594,8 @@ decode_lanman_negprot_rsp(struct TCP_Server_Info *server, NEGOTIATE_RSP *pSMBr)
                               cifs_max_pending);
        set_credits(server, server->maxReq);
        server->maxBuf = le16_to_cpu(rsp->MaxBufSize);
+       /* set up max_read for readpages check */
+       server->max_read = server->maxBuf;
        /* even though we do not use raw we might as well set this
        accurately, in case we ever find a need for it */
        if ((le16_to_cpu(rsp->RawMode) & RAW_ENABLE) == RAW_ENABLE) {
@@ -755,6 +757,8 @@ CIFSSMBNegotiate(const unsigned int xid, struct cifs_ses *ses)
        set_credits(server, server->maxReq);
        /* probably no need to store and check maxvcs */
        server->maxBuf = le32_to_cpu(pSMBr->MaxBufferSize);
+       /* set up max_read for readpages check */
+       server->max_read = server->maxBuf;
        server->max_rw = le32_to_cpu(pSMBr->MaxRawSize);
        cifs_dbg(NOISY, "Max buf = %d\n", ses->server->maxBuf);
        server->capabilities = le32_to_cpu(pSMBr->Capabilities);

diff --git a/fs/cifs/inode.c b/fs/cifs/inode.c
index 8fbbdcdad8ffa89fda849927cec56fdb7666fda6..390d2b15ef6ef9d7014e90069cdad3d2880806d3 100644 (file)
--- a/fs/cifs/inode.c
+++ b/fs/cifs/inode.c
@@ -61,7 +61,7 @@ static void cifs_set_ops(struct inode *inode)
                }
 
                /* check if server can support readpages */
-               if (cifs_sb_master_tcon(cifs_sb)->ses->server->maxBuf <
+               if (cifs_sb_master_tcon(cifs_sb)->ses->server->max_read <
                                PAGE_SIZE + MAX_CIFS_HDR_SIZE)
                        inode->i_data.a_ops = &cifs_addr_ops_smallbuf;
                else