0

I've been checking how fast is my ZFS storage server. I have some big files (>30 GB) and I've been using dd piped to /dev/null to check read speed.

When running dd for a specific file for a first time, I'm getting a consistent result of about 95 MB/s. I've used both signals and pv to monitor progress and they gave equal results.

However, when running dd for a second time, unusual thing happens:

407+0 records in
406+0 records out
425721856 bytes (426 MB, 406 MiB) copied, 4.61932 s, 92.2 MB/s
911+0 records in
910+0 records out
954204160 bytes (954 MB, 910 MiB) copied, 9.66672 s, 98.7 MB/s
1412+0 records in
1411+0 records out
1479540736 bytes (1.5 GB, 1.4 GiB) copied, 14.7018 s, 101 MB/s
12374+0 records in
12373+0 records out
12974030848 bytes (13 GB, 12 GiB) copied, 19.7579 s, 657 MB/s
12854+0 records in
12853+0 records out
13477347328 bytes (13 GB, 13 GiB) copied, 24.7491 s, 545 MB/s

What causes the sudden spike from 1.5 to 13 GB? I would think that ZFS cache is responsible (server has 64 GB RAM so it could be possible), however the server (FreeBSD) is connected to my client (OSX) over a single 1Gbps link which definitely couldn't handle 657 MB/s speed. Compression is also unlikely as the file contains nearly random data.


Edit: Sorry, perhaps I asked my question in a confusing way.

I have a FreeBSD server with ZFS filesystem. This server shares file using AFP protocol. I connect to it using desktop PC running OSX 10.10. And of course I run dd on the client.

dd if=/Volumes/NetworkShare/testfile.dat of=/dev/null bs=1048576

When running directly on the server, dd shows over 625 MB/s (which seems OK as the zpool has data striped over 8 drives).

1
  • This looks almost like the file is growing, which is inconsistent with your description of what you are doing. Please edit your question to include the exact commands you are running that produces the output in the question. For the moment, I have voted to put your question on hold as "unclear what you are asking" to ensure that people don't spend time answering the wrong questions. Should the question be put on hold before you have a chance to edit it, rest assured that editing during the "on hold" grace period will automatically nominate the question for reopen review.
    – user
    Mar 10, 2016 at 14:35

1 Answer 1

0

First of all, please include the dd command that you used to run the test. As pointed out in the comment, the file written is growing so it's likely something is not right (unless each test was copying a different file). Also to this point it matters how big chunks of data you are writing, i.e. bs option should be big enough. See this Test ZFS speed thread for an example.

Secondly, you can use the diskinfo tool to get the maximum theoretical speed of your disk, e.g.:

diskinfo -tv /dev/ada0

Thirdly, I would recommend trying different tools to get a wider picture, e.g. iozone, mentioned in the thread I posted.

Lastly, when you copy a file to /dev/null then the data never leaves your server. So you are right, the speed difference is most likely due to ZFS caching. Try to copy the file over the network and you won't be able to get more than 100Mb/s (theoretical speed of a 1Gbit link is 125Mb/s).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .