0

I am trying to manually test the read and write speed of an SSD over NVMe. The current method I am using is to mount a file system on the SSD, and read/write 20GB to a file on that filesystem in block sizes of 4KB, 32KB, 128KB, 215KB, 1MB, 64MB, 256MB, and 1GB, while recording the time of the start of the read/write and the time of completion. This process is invoked from a bash script. The bash script will attempt to run multiple "applications" by calling the below function n times, each time running the process in the background.

while [ $instCnt -le $appInstances ]
    do
    fsrw -w $blocksize /fsmnt/fs${instCnt}/usernumber1/j.j &

Here is the read function from the fsrw executable

bool perform_readop ()
{
// File descriptor.
int32_t fd = -1;

// Function status.
bool status = false;

//Zero read count
int zero_reads = 0;

// Open the file.
fd = open (fname.c_str (), O_RDONLY);

// Verify the file has been opened.
if (fd == -1)
{
    cout << get_datetime_string() << "Read open of " << fname << " failed.  Errno: " 
    << errno << endl;
}
else
{
    // Total bytes read.
    uint64_t rd = 0;

    // Elapsed time.
    struct timeval tv = { 0 };
    get_elapsed_time (&tv);

    // Notify the user that the read test has started.
    cout << get_datetime_string() << "Starting read" << endl;

    while(rd < READ_LIMIT && zero_reads < 10) {
        // Run until it is time for the test to stop.
        ssize_t readsize = read (fd, &buf[0], blocksize);
        if (readsize == -1)
        {
            cout << get_datetime_string << "Read failure.  Errno: " << errno << endl;
            zero_reads = 10;
        }
        else if (readsize == 0)
        {
            cout << get_datetime_string << "Reached EOF." << endl;
            zero_reads++;
        }
        else
        {
            rd += readsize;
        }
    }       
    // Get the elapsed time.
    get_elapsed_time (&tv);

    // Report the number of bytes read.
    cout << get_datetime_string() << "Read " << rd << " bytes in " << tv.tv_sec << "." 
    << tv.tv_usec << " seconds, bytes per second " << bytes_per_second (&tv, rd) << endl;

    // Close the file.
    close (fd);

    // Set the function return status when all read operations have
    // been successful.
    if (zero_reads < 10)
    {
        status = true;
    }
}

  return status;

}

I have ported this method from work previously done by others, and I am really not sure if this is a valid method of verifying the rate of throughput to the SSD. The results of the test, especially for the read operation, are not realistic; they are much higher than expected. Fio suggests that the throughput should be around 500MB/s read and write, but this tests records 1GB+/s write speed and read speeds near 8GB/s

4
  • I'm guessing that it's probably related to the OS caching the files as well as the SSD itself buffering the data. Accurately measuring true SSD throughput is not going to be an easy task.
    – James P
    Nov 20 '17 at 14:07
  • Had a gut feeling the nvme controller was caching. Just wanted to get some input. It seems like when I run 4 applications, and perform a write, when I go back to read, the first read of the 4 applications completes super fast. Caching might explain this. not sure how to get around this except by writing my own driver. That would not be trivial
    – John Frye
    Nov 20 '17 at 14:09
  • If you write the test files to the SSD, and reboot (with power cycling to be on the paranoid side) then on your first read after reboot your files won't be in any buffers... unmount/remount should also take care of any OS-level caching
    – xenoid
    Nov 20 '17 at 15:04
  • I tried unmounting and remounting originally in my bash script between read and write. It was throwing an error saying that there were still processes involving the partitions of the SSD. I am wondering if there is some way in bash to try to perform this operation until it is successful.
    – John Frye
    Nov 20 '17 at 15:10
0

It's unclear what your question is - you seem to be asking "why are my results not realistic (and why are they faster than fio's?)". You don't include your fio job so it's impossible to say anything about that :-( I also don't know what the fsrw program does. I'll take a stab at what remains:

Let's say we just have your program and the kernel (this is a simplification). Your program issues writes and the kernel says "yup, I've got them" the moment it queues them up in an internal buffer thus allowing your program to go on its way. Only if that internal kernel buffer is too full will your program block on the write until there is space to accept it. This means if a program is writing "small" amounts of data on a quiet system it will never end up waiting for the disk to service I/Os - we've decoupled it from the speed of the disk through the use of a buffer. Obviously this illusion can only last so long and depends on how big the buffer is, how full it gets etc.

Additionally if there is unused RAM then the kernel can use that to cache parts of disks. When I write data to the disk, if there's space the data I'm writing will not only be buffered and flushed out later but it can be kept around around after flushing finishes in case it's needed later. You can see this effect by writing a file and then checking free and typically you'll see the amount of RAM has gone down because parts of that file are being kept in cache. If I do end up reading it out of Linux's cache, the speed I achieve is typically close to the speed of RAM and the disk is untouched (if you know how to use iostat you will notice that it not reporting any/much disk I/O when this occurring).

So:

  • Normal write I/O can be buffered
  • Normal I/O can be cached
  • Reading cached data is MUCH faster than reading uncached data
  • Your read program is susceptible to the previous point

Note this is a simplification. I haven't covered things like readahead or filesystem interactions why you fsync and so on.

Additionally blocks as the kernel sends them down towards the disk may be different to the size your program submits them. You might send 16 x 4Kbyte contiguous writes to the kernel but the kernel may merge those together and send a single 64 Kbyte write down to the disk. This is usually beneficial in real life but is something you have to be aware of when doing synthetic benchmarking.

In summary, I'd guess your results are "unrealistic" because you're massively benefiting from Linux's buffering and caching. If your real workload is truly like this then the speed of your SSD isn't your bottleneck and faster disks don't help you much (so realistic is a tricky word)! You may obtain numbers closer to that of the SSD itself by ensuring your data set sizes are many (at least three) times bigger than the size of the total memory in your system or by doing I/O that can't be buffered or cached. I can't speak to why you're slower than fio runs because any answer for that is highly fio job specific and no job was included in the question.

0

Open a shell prompt.

Use the dd command to measure write speed

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync

Use the dd command to measure latency

dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 oflag=dsync

The dd command is useful to find out simple sequential I/O performance.

Use dd command to test read speed

To get accurate read test data, first discard caches before testing by running the following commands:

flush
echo 3 | sudo tee /proc/sys/vm/drop_caches

Then run the following command (after creating laptop.bin):

dd if=/dev/zero of=/tmp/laptop.bin bs=1G count=1 oflag=direct

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.