3

I noticed and tested.

If I'm tranfsering things like .pdf, .mp4, etc. of the exact same size as a .py or .pyc, the speed for the former will be ~400mb/s and the speed for the latter will be ~2mb/s?

I even tested on a folder that had files diverse types. It would always be fast when it would be transfering non-python related files, but once it starts touching python files, it drops immediately to barely 95% of what it the speed was.

Reference image: https://i.stack.imgur.com/a4vRM.jpg

What's going on behind the scenes? enter image description here

2
  • 1
    Windows Defender scans shit? Pretty sure that escsm.cpython-39.pyc isnt as big as that mp4 so that screenshot doesn't prove much
    – Gantendo
    Jul 21, 2022 at 6:55
  • from the description my guess would also go into the virus scanner direction. Trigon: can you check via task monitor if your AV is taking up hard drive time? during those transfers? Taht could be a very strong indication into that direction
    – Thomas
    Jul 21, 2022 at 14:18

2 Answers 2

1

Although not clarified in the question, the problem is likely due to the data not being cached and instead synchronized when copying to a removable or network device. This is normal behavior due to both the overhead of uncached data and synchronized transfers to these kinds of devices. It can also be slower even to another local disk due to API and filesystem overhead that is incurred for each individual file. The problem is that when each file is transferred individually and not in parallel, this problem is more pronounced.

See here:

https://superuser.com/a/1448143/373021

The problem isn't that they are python files, but that they are small files, possibly even on a fragmented device if it's a spinning disk.

You can verify this by renaming the files to something other than .py and you'll see the same result. You'll notice that, while copying .mp4 files is much faster, renaming the .py files to .mp4 does not speed up the transfer. Thai is because typical .mp4 videos are large single continuous files, which do not suffer from any of the aforementioned problems.

1

This is what's going on behind the scenes:

You are copying 6430 files and after some time the copy slows down. As you're seeing at that time python files being copied, the natural conclusion is the python files are slow to copy.

It's possible that the problem is with your antivirus, that insists on checking the files one-by-one, but this is somewhat less likely.

I believe that this behavior is because what you're actually seeing is how fast do the caches fill-up on Windows and in the target (if not on the same computer).

Windows will start by reading from the disk at its speed to copy it to the target at a lesser speed. At this stage the speed you're seeing is basically that of reading from the disk.

When the Windows memory cache fills up, the speed will drop to that of writing to the target, since cache memory needs to be liberated in order to read more from the disk. The speed then drops to the writing speed, which is always less than reading speed, especially for small files.

In addition, you're using Explorer for the copy, which is one of the slowest products I know in this domain.

4
  • why do you think that the AV can't be the cause? From my personal experiences I've seen a few AVs that check every file that comes into memory (even by just copying it), if it has specific exensions like .py,...
    – Thomas
    Jul 21, 2022 at 14:16
  • @Thomas: Normally an antivirus does not check text files (unless in sensitive Windows folders or part of a web-app). I agree that it's still possible, as I don't know which antivirus is used.
    – harrymc
    Jul 21, 2022 at 16:09
  • @harrymc Most AV product leave TXT files alone, but PY files are not considered text-files, but executable scripts and will get scanned every time.
    – Tonny
    Jul 22, 2022 at 9:30
  • @Tonny: Depending on the AV and its settings, as there is a long list of possibly dangerous text files: bat, cmd, vbs, php etc.
    – harrymc
    Jul 22, 2022 at 9:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .