Assuming that you run those conversions from and to a rotating disk, you are very likely to make a CPU-bound job into a disk-bound one.
A conversion process consist of three tasks:
- Read the original file from disk
- Do some calculation on the data
- Write the result file to the disk
A rotating disk is good at sequential reads or writes, but extremly bad at random IO - so even a single conversion can be hampered by the concurrency between 1. and 3. This implies that a conversion from one physical disk to another is likely to be faster than a conversion from a disk to itself.
If you now multiply this concurrency by three, you are very likely to end up in a scenario, where the seek and wait-on-rotation times of the disk by far outweigh the actual read times - this can easily lead to throughput going down by orders of magnitude: It is not uncommon for a disk, that can reach over 100MB/s sequential read, to reach less than a single MB/s random reads.
A usually seen pattern is very fast inital performance, while the writes are buffered in RAM, but dropping of a cliff, when the cache is full and the writes really need to hit the disk.
- First of all get rid of spinning rust - it is 2020.
- If that is no option, then try to limit IO concurrency by using different disks for read and write. The best way might be to create a RAM disk as a target device (as usual in the broadcast industry). In fact since RAM is so cheap it might be a good idea to convert from a RAM disk into a RAM disk.
- Carefully chose the number of concurrent conversion jobs to find the sweet spot between IO saturation and CPU/GPU saturation.