4

My machines all run SSDs and 1GB network connections. I can move massive VHDs over the network very quickly. However, I am not able to move 128GBs worth of smaller files from one machine to another quickly at all.

Are there features built within Windows 10/Server 2012 that can be used other then doing a drag and drop approach?

I was looking under the features that you can install and there are some file moving related options but are they of any benefit to me?

I even checked network traffic and there is no saturation or load on the machines to cause this slow effect of 20/kbs.

EDIT: Due to the amount of files and the locations that they all came from, linux, mac, ect... some of the file names have characters in them that prevent the files from being sent to a zip file and stopping the whole process. Is there a way around that?

  • 1
    Processing large number of small files can be a pain regardless of OS. One approach is to create a large one-file archive that can be moved relatively quickly. In Linux I would tar (and gzip maybe) and ssh in a pipe. This skips saving the archive as a file – an obvious pain with total size of 128 GB. Unfortunately I don't know similar Windows solution. Maybe with Bash on Ubuntu on Windows you could utilize the Linux approach. – Kamil Maciorowski Jan 5 '17 at 16:45
  • 1
    Yes, SAMBA is the protocol Windows uses for network files and it is very inefficient. Zip the files with 7Zip and copy them. It will be faster even with the extra step. Or move them onto MicroSD cards and transfer that way. If you are an expert, you can set up a SFTP server and use WinSCP, or install Unison. – Chloe Jan 6 '17 at 2:09
2

Over the years, my mentors always emphasized to me that there were a few factors that could potentially contribute to this phenomenon such as indexing and security software, but ultimately it comes down to the operating system negotiating each file transfer.

As an example, if you are moving one very large AVI file that only needs to be done once. On the other hand, if you are copying hundreds of JPG files, that will create quite a bit more overhead.

If you are looking for a method to help mitigate the problem, you might trying putting a bunch of small files into one larger one using 7-Zip or something similar. It probably won't make a dramatic difference, but a tangible one nonetheless.

1

Transferring one large file is always faster than transferring many smaller files, assuming that the total size of the smaller files is equal to the size of the big file. You will notice this difference in copying from CD or DVD, from network and on traditional HDDs.

There are several factors contributing to this slowness. The following is a list of things that happen once for the large file but several times for many small files.

  1. Creating file's entry in the file system
  2. Consulting the file system to locate the file on disk
  3. Scanning the file for viruses (real-time protection feature of the AV)
  4. Creating folders and subfolders pertaining each file
  5. Checking user access rights for the file

I was looking under the features that you can install and there are some file moving related options but are they of any benefit to me?

In this case, not really. Your best option to archive the files you want to transfer into a single ZIP, RAR or 7z file. Even with no compression, your transfer will be much faster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.