2

Taking in consideration the time it would take to decompress the zip folders once copied to a flash drive, what would be faster? Copying the compressed folders to a usb flash drive then decompress them on the flash drive or just copying the uncompressed folders to the flash drive.

Don't take in consideration the time it takes to compress the files since the files are already compressed before copying. I can copy either from a USB HDD or from an SSD to a USB flash drive (both supporting USB 3.0).

In this specific case I want to transfer 9 zip files which have in total 115,518 files, most of the files are really small image files (the 9 files in total amount to 15Gb uncompressed and 10 Gb compressed).

  • 2
    SATA3 is significant faster then USB 3. However Read and Write speeds are faster on Flash. It entirely depends on the archive – Ramhound Sep 10 '17 at 21:16
  • 1
    Try it on your hardware and find out? – Andrew Henle Sep 10 '17 at 21:22
  • 1
    There are too many variables here to find out. Decompressing ZIPs is pretty fast on modern CPUs, so reading a small amount of data and decompressing will be faster than reading a larger amount of uncompressed data. (On very old computers, before 1995, that wasn't necessarily true due to slow CPUs being slow at decompressing.) So using a ZIP file will save time. Will it save more time than it takes for you to do an extra copy? That I don't readily know offhand. Hardware speeds may be a significant factor in those calculations. So is your speed, possibly. – TOOGAM Sep 10 '17 at 21:33
  • 2
    The fastest way is obviously to leave the ZIP file wherever it is and unpack it directly to the final destination. // Keep in mind that exFAT is super terrible with small files. – Daniel B Sep 11 '17 at 8:10
2

Taking in consideration the time it would take to decompress the zip folders once copied to a flash drive, what would be faster? Copying the compressed folders to a usb flash drive then decompress them on the flash drive or just copying the uncompressed folders to the flash drive.

Probably the latter.

Remember that decompression and other such things are done by the computer's CPU. You can copy data into a drive (be it flash or HDD), and you can copy data out, but you cannot tell the drive itself to decompress it – or do anything else, really.

So your first plan would involve:

  1. Copying 10 GB of data from disk to flash
  2. Reading those 10 GB back from flash to RAM
  3. Decompressing it
  4. Writing 15 GB of the decompressed data from RAM to flash
  5. Deleting the useless compressed files left after step 1

The second:

  1. Reading 10 GB directly from internal disk to RAM
  2. Decompressing it
  3. Writing 15 GB of decompressed data to flash

Notice how the first plan involves twice as much reading&writing. But not only that: in reality, steps 2-3-4 wouldn't happen in order but in parallel – which makes the process faster when reading from one drive and writing to another, but much slower when the same drive is doing both reads and writes.

  • Both plans wouldn't involve putting the entire 10 GB into memory in one big chuck, it would be done in smaller chunks, even if you have 10 GB of memory (only way that wouldn't be the case is if you had 10 GB compressed files within the archive itself. Compression programs are pretty darn efficient. – Ramhound Sep 10 '17 at 22:30
  • @Ramhound However, in no configuration would it be possible for any drive that time(Writing_seq + Reading_seq + Writing_rand) < time(Writing_rand). That is: as long as drives don't perform better the more tasks they have to run simultaneously. The resulting files are 15GB of size in both configurations - only one time, they get copied straight away, while the other time, you additionally write & read 9GB of sequencial file(s). – flolilo Sep 10 '17 at 22:35
1

What would be faster, transferring a zipped file to flash drive then decompress there or transferring the unzipped files?

Unquestionably the correct answer is "transferring the unzipped files".

Of course transferring a zipped archive of all the files would be faster, but you have stipulated the additional step of "then decompress there".

Apparently you are not aware that you cannot "decompress" locally on the drive or storage device.
The source archive file has to be read (from where ever it is stored) and then decompression will be performed in main memory by the CPU.

So in order to unzip the file already transferred to the flash drive, the file has to be read from the flash drive, uncompressed, and then the individual files written to the flash drive.

That last step (of writing the individial files) alone is equivalent to the latter choice. So the additional steps of first writing the zip file(s) and then reading them back (but possibly optimized by caching) should made your first choice the longer operation.



Another way of looking at your question is assuming some amount of concurrency among operations and guessing at a timelime.

So for case 1, the copy of the zip file to the flash drive requires read operations concurrent with write operation (but with some block delay).
Writing also requires more time to perform than reading (e.g. see Why does copying the same amount of data take longer if spread across many separate files?).
Once the copy is complete then it can be read, decompressed and written back to the flash drive.
Despite the concurrency of the reading, decompressing, and writing operations, the overall timeline is dominated by the time to write all of the uncompressed files.

RRRRRRRRR  
 WWWWWWWWWW  
           rrr    rrr   rrr  
            DDDD   DDDD  DDDDD
              WWWWWWWWWWWWWWWWWWWWW  
time-->

Case 2 is a straightforward read, decompress and write.
This case should require the same amount of time to write all of the uncompressed files.

RRR    RRR   RRR  
 DDDD   DDDD  DDDDD
   WWWWWWWWWWWWWWWWWWWWW

How can you expect Case 1 to complete faster than Case 2 when the the first stage of read operations is for the same amount of data from the same device, and the last stage of writes is for the same amount of data to the same device
and
case 1 is handicapped by having to first perform a copy?

0

If you want to unzip to the same drive - don't bother zipping at all. If you will unzip to the other drive - consider zipping, as it will make less IO calls for the filesystem (less data and less file entities)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.