I've been checking how fast is my ZFS storage server. I have some big files (>30 GB) and I've been using dd
piped to /dev/null
to check read speed.
When running dd
for a specific file for a first time, I'm getting a consistent result of about 95 MB/s. I've used both signals and pv
to monitor progress and they gave equal results.
However, when running dd
for a second time, unusual thing happens:
407+0 records in
406+0 records out
425721856 bytes (426 MB, 406 MiB) copied, 4.61932 s, 92.2 MB/s
911+0 records in
910+0 records out
954204160 bytes (954 MB, 910 MiB) copied, 9.66672 s, 98.7 MB/s
1412+0 records in
1411+0 records out
1479540736 bytes (1.5 GB, 1.4 GiB) copied, 14.7018 s, 101 MB/s
12374+0 records in
12373+0 records out
12974030848 bytes (13 GB, 12 GiB) copied, 19.7579 s, 657 MB/s
12854+0 records in
12853+0 records out
13477347328 bytes (13 GB, 13 GiB) copied, 24.7491 s, 545 MB/s
What causes the sudden spike from 1.5 to 13 GB? I would think that ZFS cache is responsible (server has 64 GB RAM so it could be possible), however the server (FreeBSD) is connected to my client (OSX) over a single 1Gbps link which definitely couldn't handle 657 MB/s speed. Compression is also unlikely as the file contains nearly random data.
Edit: Sorry, perhaps I asked my question in a confusing way.
I have a FreeBSD server with ZFS filesystem. This server shares file using AFP protocol.
I connect to it using desktop PC running OSX 10.10. And of course I run dd
on the client.
dd if=/Volumes/NetworkShare/testfile.dat of=/dev/null bs=1048576
When running directly on the server, dd
shows over 625 MB/s (which seems OK as the zpool has data striped over 8 drives).