10

I've noticed that whenever I copy or move large files from my SSD, which I use as the system drive, to my HDD or to an external hard or flash drive, the speed graph shown by Windows always looks the same: the transfer speed starts at around 450 MB/s, and after a few seconds drops down to somewhere between 90 and 130 MB/s and remains stable until the end of the copy/move operation.

Transfer speed graph

This sparked my curiosity, so I decided to figure out what is the cause of this. Some of my thoughts were these:

Maybe that is the actual speed at which the transfer happens

Doubtfully. While the 450 MB/s speed matches the rated speed of my SSD, considering I also have some other disk reads/writes going on in the background, there is no way a 7200 rpm hard drive is capable of keeping up with it, as the 130 MB/s speed I get later on is also the most I can expect from it. So, where does the extra data go?

The extra data is being stored in the hard drive's cache memory

This makes a bit more sense, but if I take into account the duration of the higher transfer speed, my hard drive's cache would have to be over 3 GB in size, which it definitely isn't. What else could it be?

The extra data is being stored in the RAM

This makes sense. My RAM is the only other part of my system which can match the speed of my SSD, and I have plenty of it. Let's check this theory!

I open up the Task Manager, and take a look at the Performance tab. Memory usage is stable at 3.7 GB. Then I start another 15 GB file transfer. Memory usage starts rising, and stops at 5.3 GB just as the transfer speed drops to 130 MB/s. It remains the same until the end of the file transfer(the transfer dialog closes), and then slowly drops back to the 3.7 GB level it was before the transfer.

So, my last theory is true. Further confirmation is the fact that the extra used memory is marked as Modified

Modified.

What's the point?

My question is, what is the purpose of doing this? While I don't mind having some of my RAM used by file transfers, as even during the heaviest of my multitasking sessions I've never seen its usage go over 70%, what is the benefit of storing 1.6 GB of data which you won't be doing any kind of processing on in your RAM?

I don't see any benefit from the data integrity standpoint, as you're merely copying the files, and in the case of a power failure neither the RAM or the HDD will be particularly successful in retaining the data in transfer.

I could see the benefit being that the source disk(the SSD) is quickly freed up, so that if another process needs to perform lots of read/write operations on it, it can do so without the file transfer impeding it, but if that is the case, why not go ahead and load all 15 GB at max speed into the memory?

Also, this process misleads the user, as the file transfer keeps going even after the transfer dialog closes, because some of the data is still being copied from the memory to the hard drive. This could cause a user to plug out a removable drive while data is still being written to it, possibly corrupting the removable drive, cause not everyone bothers with safely removing hardware.

Keep in mind I haven't thoroughly tested this with removable drives, as Windows might be handling them differently, making my last point invalid.

8
  • 3
    The point is a speed up by caching disk information in free memory. Should you need to file again then there is no need to read it from a slow device (and yes, even an SSD is still slow compared to RAM). So almost any modern OS reads and writes to a cache, and that cached info get flushed (for writes) or free'd (for reads) whenever the OS needs more free memory.
    – Hennes
    Oct 18, 2015 at 15:40
  • Oh, and unplugging an external drive can be done with cached but unwritten data. 'safely remove this device' might take a while though (e.g. until all information is written). This is very noticeable when I copy from my laptops internal SSD to an external USB HDD. But as long as you follow the normal method (which should always involve safely remove this device) all is safe.
    – Hennes
    Oct 18, 2015 at 15:42
  • @Hennes But the OS(Windows in this case) completely removes all of the data from the RAM as soon as the file transfer is complete, and what is the chance that you will need to access the file in the middle of a transfer operation, and that you will be accessing the exact part which is loaded into the memory at that moment? Doesn't really make sense to me.
    – akukas
    Oct 18, 2015 at 15:49
  • "I've seen SCSI host adapters that can perform drive to drive transfers (without the involvement of CPU and RAM)...These are the exceptions, not the rule or in common use." Exactly, they'll be adapters with their own CPU and RAM to handle those transfers. :) Oct 18, 2015 at 17:14
  • So, where does the extra data go? - there is no "extra data", what you see is read/write speed, not amount of data. No need/use to read more data than can be accepted by the destination device. Copy data to /dev/null for read speed from a single device, copy data from /dev/zero for writing to a single device.
    – Hannu
    Oct 18, 2015 at 17:20

2 Answers 2

2

Windows memory management is a complex thing. As you see it has different behavior with different devices.

The different operating systems has different memory management.

Your question was very interesting. I am sharing a MSDN page which explains a part of the memory management in windows and more specifically "Mapped Files"

It's documentation for software developers, but Windows is software too.

One advantage to using MMF I/O is that the system performs all data transfers for it in 4K pages of data. Internally all pages of memory are managed by the virtual-memory manager (VMM). It decides when a page should be paged to disk, which pages are to be freed for use by other applications, and how many pages each application can have out of the entire allotment of physical memory. Since the VMM performs all disk I/O in the same manner—reading or writing memory one page at a time—it has been optimized to make it as fast as possible. Limiting the disk read and write instructions to sequences of 4K pages means that several smaller reads or writes are effectively cached into one larger operation, reducing the number of times the hard disk read/write head moves. Reading and writing pages of memory at a time is sometimes referred to as paging and is common to virtual-memory management operating systems.

Unfortunately we can't easy figure how Microsoft implements the Read/Write - it isn't open source.
But we know that it has very different situations:

From      To
==================
SSD       HDD
HDD       Busy SSD ??
NTFS      FAT
NTFS      ext4
Network   HDD
IDE0slave IDE0master // IDE cable support disk to disk transfer.
IDE       SATA // in this case you have separated device controllers.

You get the point... A hdd may be bussy, the file systems may be different (or may be the same)...

For example: dd command in linux copying data "byte by byte" - It's extremely fast (because the heads of both HDDs moving sync), but if the file systems are different (with different block sizes for example) - the copied data will not be readable because the file system has different structure.

We know the RAM is much much faster than HDD. So if we have to do some data parsing (to fit the output file system) it will be better to have this data in the RAM.

Also imagine you coping the file directly from-to.
What's happening if you overload the source with other data flows? What about the destination?
What if you almost doesn't have free RAM in this moment?
...

Only Microsoft engineers know.

5

Why do file transfers between drives use RAM?

Because I/O operations are (almost always) between a peripheral and RAM.
So a file copy is really two disk operations: a read (to RAM) and then a write (from RAM).

Some systems can perform peripheral-to-peripheral operations (and therefore not require a buffer in RAM). I've seen SCSI host adapters that can perform drive-to-drive transfers (without the involvement of CPU and RAM by using an on-board processor and RAM/FIFO). I've seen DMA controllers that can perform peripheral-to-peripheral transfers. These are the exceptions, not the rule nor in common use.

Note that error handling is doubly complicated when using peripheral-to-peripheral transfers, and is therefore rarely used by an OS for arbitrary I/O operations even when the hardware is available.

My question is, what is the purpose of doing this?

The source data (the file being read/copied) has to be read into RAM simply because that is the nature of computer archtecture (combined with the use of DMA).
Back in the day when RAM was scarce and OSes were not as sophisticated, the typical choice was to perform these transfers with a buffer as small as possible (e.g. just one sector or block), or a buffer that would optimize speed (e.g. a multi-sector transfer).
Nowadays with relatively large memory and sophisticated algorithms, the typical OS will try to use any/all of free/unused memory for the buffer to read this source data.

After that source data has been written to the destination, that data can be discarded. But if there is no demand for this memory (i.e. it will remain unused), then there is no imperative to discard this data.
And if the data has been tagged or cataloged to identify the file it came from, then this data can be cached for possible reuse.

Note that this cache of files is essentially free; there was no burden of additional I/O on the OS to obtain this cache. The only cost is maintaining the catalog of the contents, which can be offset when there's a cache hit (and save the I/O of a reread).


So the caching of files is a free consequence of normal reading of those files. It temporarily uses memory that would otherwise be idle and unused. The overhead in the OS for maintaining this cache is typically minimal and is paid back when cache hits occur.


Also, this process misleads the user, as the file transfer keeps going even after the transfer dialog closes, because some of the data is still being copied from the memory to the hard drive.

That is a trade-off between system availability to the user versus guaranteed writes.
A program can be written to perform blocked synchronous writes, which would wait for each physical write operation to complete before the program continued. Or the program, at judicious points, could wait for the physical write operations to complete by invoking a sync() or similar system call.

In a write (or copy) operation, the modern OS will try to be available to the user as soon as all write operation have been at least queued up (meaning that they may not have completed).
This facilitates multitasking. If you wanted to do something else that didn't even involve that drive, why should you have to wait for operations on that drive to finish to do that something else?
The trade-off is that you have to be an educated user, and be cognizant that (storage) devices need to be properly unmounted.

This availability feature has nothing to do with the above mentioned file cache feature. If it were available/possible, you could disable one without affecting the other.


Note
A data transfer could use no RAM buffer if:
1. programmed I/O (using the CPU) is performed for both the input and the output operations,
AND
2. the input and output have matching data transfer rates and transfer sizes,
AND
3. both devices are character-oriented and not block devices. (This would exclude disk drives.)

However an OS will rarely choose to use programmed I/O when DMA or bus mastering is available, nor is likely to be programmed to handle the double complexity of a paired input+output transfer.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .