24

I tired running 8 commands in parallel to fully utilize the CPU and speed up Video conversions, something like this:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/b.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/c.mp4 > /dev/null 2>&1  &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/d.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/e.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/f.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60  -f mp4 /mnt/g.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4  /mnt/h.mp4 > /dev/null 2>&1 &

2 to 3 of them get stopped. Why does this happen? Is this a FFmpeg limitation? I have tried this on 16 core and 4 core machines, EC2 c1.xlarge and cc2.8xlarge. Same behavior. I have tried complicated commands and simple ones, still same, 2 or 3 get stopped.

4
  • 3
    To my knowledge there shouldn't be anything in FFmpeg itself stopping you from doing so. Could be a shell issue?
    – slhck
    Commented Jan 18, 2013 at 11:28
  • What if I have different inputs for each command? How can I make them independent, because if one stops they all stop.
    – Samson
    Commented Aug 9, 2013 at 9:39
  • Did you find any parameter for using multi processors?!
    – Dr.jacky
    Commented Aug 19, 2015 at 4:45
  • what do you mean by "get stopped" here? bash shows them as paused?
    – rogerdpack
    Commented Jan 31, 2016 at 4:33

3 Answers 3

26

In answering the original question, as to why some of the jobs stop, ffmpeg on the command line is interactive. It's constantly reading input on the command line. In order for you to run all of them in the background you should change this:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 > /dev/null 2>&1 &

to this:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 </dev/null > /dev/null 2>&1 &

the addition of </dev/null tells ffmpeg to not look for input, and all of your jobs should run in the background.

4
  • 1
    What is this: 2>&1
    – Dr.jacky
    Commented Aug 10, 2015 at 6:12
  • Thank you very much! I had the very same problem and this worked like a charm, please accept!
    – fr_andres
    Commented Jun 21, 2017 at 23:23
  • 4
    @Mr.Hyde Late to the party I know, but 2>&1 tells the error stream to go to the same place as the stout stream - so /dev/null. It's shorthand for > /dev/null 2>/dev/null.
    – berry120
    Commented Feb 5, 2018 at 17:26
  • 1
    @Dr.jacky An answer to your comment was posted to your alter-ego Mr. Hyde. Commented Jul 25, 2020 at 17:58
9

Just a thought on your command: a much easier way of hammering the machine with a single command is concatenating everything into a single line:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 \
 -f mp4 /mnt/b.mp4 \
 -f mp4 /mnt/c.mp4 \
 -f mp4 /mnt/d.mp4 \
 -f mp4 /mnt/e.mp4 \
 -f mp4 /mnt/f.mp4 \
 -f mp4 /mnt/g.mp4 \
 -f mp4  /mnt/h.mp4 > /dev/null 2>&1

You might want to add (depending on your ffmpeg version) a flag to tell the server to use all available processors

-threads 0

For reference: http://ffmpeg.org/trac/ffmpeg/wiki/Creating%20multiple%20outputs

5
  • 1
    I have tried this, it does not use all the processors efficiently. The load remains at 20-30%, the hyper threads don't get used at all.
    – d33pika
    Commented Feb 7, 2013 at 1:25
  • 2
    Maybe it has something to do with EC2 machines? I just run the above command on a 16 xeon core machine with 16 outputs and this is what I get: htop screenshot
    – NublaII
    Commented Feb 8, 2013 at 16:09
  • Are you on the latest version of FFmpeg?
    – d33pika
    Commented Apr 3, 2013 at 2:20
  • @NublaII What is > /dev/null 2>&1 !?
    – Dr.jacky
    Commented Aug 23, 2015 at 11:01
  • 1
    @Mr.Hyde It hides the standard output from you and shows you error outputs. Read this: xaprb.com/blog/2006/06/06/what-does-devnull-21-mean Commented Sep 7, 2016 at 22:52
1

I can get all CPUs engaged, each at 100% utilization, using:

find . -name "*.mp4" -print0 | parallel -0 ffmpeg -i {} {.}.mp3 -hide_banner' \;

parallel options:

  1. {} is replacement string (file name) from input source (find)
  2. {.} is file name without extension

You can run the command inside the folder containing the media files. You can also run it to process media files in multiple sub folders.

I run this often on Ubuntu on Windows 10 WSL, successfully processing few 1000 files at a time. I've never had a problem (verified count of input and output files)

to install 'parallel':

sudo apt-get install parallel

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .