Tl;dr – We can't find the reason for the limited write speed of 60 MB/sec to our NAS via SMB and AFP from two different Mac clients. In comparison: An old Windows 7 laptop in the same network writes steady 100 MB/sec.
If you read this question for the first time, please skip to the Update 4 section. rsync
is the main reason for the low speed, even though we don't understand why (for a single file!).
Original Question: Find speed bottleneck SMB3/NAS with Mac OS 10.11.5 and above
We tested via rsync --progress -a /localpath/test.file /nas/test.file
on macOS and the copy info of Windows.
The NAS is a DS713+ running their current DSM 6.0.2 (tested with 5.x too), with two HGST Deskstar NAS SATA 4TB (HDN724040ALE640) in RAID1 with only gigabit ethernet components and new ethernet cables (at least Cat5e).
Mac clients first only made 20 MB/sec. But applying the signing_required=no
fix (described here) pushed the write speed to 60 MB/sec via SMB2 and SMB3. AFP also delivers around 60 MB/sec. Result varies around 5 MB/sec depending on protocol and (Mac) client.
What we've already tried:
Network
- Tested network performance via iperf3. Result: 926 Mbit/s. Looks good.
- Tried Dual Link Aggregation/Bonded network interfaces. No change.
- Increased MTU to 6000 and 9000. No change.
- Checked all cables. All fine at least Cat5e, in good condition.
Disks
- Checked S.M.A.R.T. Looks healthy.
- Tested write speed directly to disk with
dd if=/dev/zero of=write.test bs=256M count=4
with variousbs
andcount
settings (128/8, 512M/2, 1024/1). Result: around 120 MB/s (depending on block size/count)
SMB/AFP
Benchmarked SMB2, SMB3 and AFP against each other. About equal.
See update below: Used wrong method to rule out the SMB implementation of macOS. SMB on Windows is faster, new SMB settings coming with macOS 10.11 and 10.12 may be the reason.- Tried to tweak the SMB settings, including the socket options (following this instruction)
- Tried different version of delayed ack settings and
rsync --sockopts=TCP_NODELAY
(comments)
No significant change of the write speed. We double checked that the config was really loaded and we were editing the right smb.conf.
System
- Watched CPU and RAM load. Nothing maxes out. CPU around 20%, RAM about 25% during transfer.
- Tested the same NAS with DSM 5.x.x in a nearly out-of-the-box setup. No additional software installed. Note: We have two of these in different locations. They are in sync via Synology's CloudSync. Same result.
- Deactivated everything unnecessary which could draw system resources.
We think this is a rather default setup, no fancy adaptions, clients or network components. According to the metrics Synology publishes the NAS should perform 40 MB/s to 75 MB/s faster. But we just can't find the bottleneck.
Clients/NAS
The Mac clients are a MacPro 5,1 (standard wired NIC, running 10.12.3 (16D32)) and a MacBookPro10,1 (Thunderbolt network adapter, running 10.11.6) only about 2m cable away from the NAS, running over the same gigabit switch as the Windows laptop in the test.
We have two of these NASes in different locations and the results are identical. The seconds NAS is more or less factory default (not even 3rd party software installed). Just two RAID1, EXT4 formatted disks syncing to the other NAS via Synology CloudSync. We've gone as far as directly connecting to the NAS without the switch, same result.
Important Update
The method used to rule out the SMB implementation of macOS/OS X was wrong. I tested it via a virtual machine, assuming it would use its own version of SMB, but obviously the traffic gets handed to macOS, running through its version of SMB.
Using a Windows laptop I've now been able to achieve an average of 100 MB/s. Indicating the SMB implementation/updates coming with 10.11 and 10.12 may cause the poor performance. Even if signing_required
is set to no
.
Would be great if someone could point out some further settings that may have changed with the updates and could affect the performance.
Update 2 – new insights
AndrewHenle pointed out in the comments that I should investigate the traffic in detail using Wireshark for more insight.
I therefore ran sudo tcpdump -i eth0 -s 65535 -w tcpdump.dump
on the NAS, while transferring two test files one with 512 MB and one with 1 GB. And inspected the dump with Wireshark.
What I found:
- Both OS X and Windows seem to use SMB2 although SMB3 is enabled on the NAS (at least according to Wireshark).
- OS X seems to stick with the MTU. The packets have 1514 bytes leading to way more network overhead and sent packets (visible in the dumps).
- Windows seems to send packets up to 26334 bytes (if I read the data correctly! Please verify.) even if the MTU shouldn't allow that, since it's set to 1500 on the NAS, the maximum setting would be 9000 there (Synology also uses the 1500 setting in their tests).
- Trying to force macOS to use SMB3 by adding
smb_neg=smb3_only
to the /etc/nsmb.conf didn't work or at least didn't lead to faster transfers. - Running
rsync --sockopts=TCP_NODELAY
with various combinations of TCP delayed ack settings (0 to 3) had no effect (Note: I ran the tcpdump with the default ack setting of 3).
I've created 4 dumps as .csv files, 2 while copying 512 MB (test-2.file) and 2 while copying 1024 MB (test.file). You can download the Wireshark exports here (25.2 MB). They are zipped to save space and named self-explanatorily.
Update 3 – smbutil output
Output of smbutil statshares -a
as requested by harrymc in the comments.
==================================================================================================
SHARE ATTRIBUTE TYPE VALUE
==================================================================================================
home
SERVER_NAME server-name._smb._tcp.local
USER_ID 502
SMB_NEGOTIATE SMBV_NEG_SMB1_ENABLED
SMB_NEGOTIATE SMBV_NEG_SMB2_ENABLED
SMB_NEGOTIATE SMBV_NEG_SMB3_ENABLED
SMB_VERSION SMB_3.0
SMB_SHARE_TYPE DISK
SIGNING_SUPPORTED TRUE
EXTENDED_SECURITY_SUPPORTED TRUE
LARGE_FILE_SUPPORTED TRUE
OS_X_SERVER TRUE
QUERYINFO_NOT_SUPPORTED TRUE
DFS_SUPPORTED TRUE
MULTI_CREDIT_SUPPORTED TRUE
--------------------------------------------------------------------------------------------------
Note on this: I'm sure SIGNING_SUPPORTED
being true
here doesn't mean the setting in the config doesn't work. But only that it is supported by the NAS. I've triple checked that changing the signing_required
setting in my config has an effect on the write speed (~20 MB/s when tuned on, ~60 MB/s when off).
Update 4 – Samba Wars: A New Hope
It feels somewhat embarrassing, but the main problem here – again – seems to be the measurement.
Turns out rsync --progress -a
costs about 30 MB/s of writing speed. Writing with dd
directly to the SMB share and using time cp /local/test.file /NAS/test.file
are faster at about 85-90 MB/s and apparently the fastest way to copy is the macOS Finder at around 100 MB/s (which is also the method hardest to measure, since there is no timing or speed indicator – who needs that, right? o_O). We measured it by first copying a 1 GB file and then a 10 GB file, using a stopwatch.
What we've tried since the last update of this question.
- Copy from Mac client to Mac client. Both have SSDs (MacPro writes with 250 MB/s to own disc, MacBook Pro with 300 MB/s). Result: A meagre 65 MB/s via
dd
writing from MacBook Pro to MacPro (rsync
25 MB/s). Seeing the 25 MB/s was the moment when we started questioning rsync. Still 65 MB/s are extremely slow. So the SMB implementation on macOS seems… well, questionable. - Tried different ack settings with dd and cp – no luck.
- Finally we found a way to list all the available nsmb.conf options. It is a simple
man nsmb.conf
. Caution the online version is outdated!
So we tried a few more settings, among them:
notify_off=yes
validate_neg_off=yes
read_async_cnt=16
write_async_cnt=16
dir_cache_async_cnt=40
protocol_vers_map=4
streams=no
soft=yes
Note: smb_neg=smb3_only
is – as I already expected – not a valid setting. protocol_vers_map=4
should be the valid equivalent.
Anyway, none of these settings made a difference for us.
New questions at a glance
Why is rsync such an expensive way to copy one (1!) file. There isn't much to synchronize/compare here, is there? The tcpdump doesn't indicate possible overhead either.
Why are
dd
andcp
slower than the macOS finder when transferring to a SMB share? It seems when copying with Finder there are significantly fewer acknowledgements in the TCP communication. (Again: The ack setting e.g.delayed_ack=1
made no difference for us.)Why does Windows seem to ignore the MTU, sending significantly larger and therefore fewer TCP packets, resulting in the best performance, compared to everything possible via macOS.
This is what the packets look like from macOS (constantly 1514)
"TCP","1514","[TCP segment of a reassembled PDU]"
"TCP","66","445 > 56932 [ACK] Seq=6603 Ack=35239 Win=4505 Len=0 TSval=520980697 TSecr=650208630"
And this coming from Windows (up to 26334, varying in size)
"SMB2","1466","Write Request Len:65536 Off:196608 File: test.file"
"TCP","26334","[TCP segment of a reassembled PDU]"
"TCP","7354","[TCP segment of a reassembled PDU]"
"TCP","54","445 > 49220 [ACK] Seq=6831 Ack=267030 Win=4074 Len=0"
You can download full .csv here (25.2 MB), the file names explain what has been copied (OS, transfer method and file size).