0

I have a set of 4-5 machines which almost exclusively encode video all day, and sometimes all night too. I've searched many times to find ways to optimize them for this purpose, but not had any success.

My premise: I should be able to use nearly all of my cpu for video encoding, if and only if, I am running multiple encoding processes simultaneously, which I am.

The Problem: ( from 'top' )

Tasks: 382 total,   2 running, 380 sleeping,   0 stopped,   0 zombie
%Cpu(s): 51.6 us, 36.8 sy,  0.0 ni,  9.6 id,  0.0 wa,  2.0 hi,  0.0 si,  0.0 st
KiB Mem:  16425680 total, 16171664 used,   254016 free,    30492 buffers
KiB Swap: 16769020 total,   406056 used, 16362964 free. 15198692 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                         
31308 root      20   0  486568  26000   5560 S  92.2  0.2 188:10.72 avconv                                                                                                                          
10340 root      20   0  475832  19920   5532 S  91.2  0.1  13:40.42 avconv                                                                                                                          
12547 root      20   0  473832  23672   5528 S  89.9  0.1   5:40.23 avconv                                                                                                                          
 8179 root      20   0  477132  21200   5560 S  89.2  0.1  71:50.15 avconv                                                                                                                          
 9059 root      20   0  472776  17960   5540 S  89.2  0.1  48:12.62 avconv                                                                                                                          
 9798 root      20   0  477812  16976   5532 S  88.9  0.1  25:23.15 avconv                                                                                                                          
 8725 root      20   0  476068  21696   5560 S  87.2  0.1  62:12.02 avconv                                                                                                                          
12657 root      20   0  472332  22012   5524 S  86.9  0.1   6:21.28 avconv                                                                                                                          
12915 root      20   0  473732  41888   5516 S  86.6  0.3   5:22.63 avconv                                                                                                                          
 8578 root      20   0  476840  24100   5536 S  85.6  0.1  63:17.07 avconv                                                                                                                          
 7822 root      20   0  472740  18940   5536 S  85.2  0.1  75:22.04 avconv                                                                                                                          
 9948 root      20   0  485420  18116   5560 S  84.9  0.1  22:18.23 avconv                                                                                                                          
 7356 root      20   0  476744  20084   5560 S  84.6  0.1  92:31.91 avconv                                                                                                                          
 9425 root      20   0  471096  19684   5552 S  82.9  0.1  37:48.28 avconv                                                                                                                          
12464 root      20   0  475004  24304   5532 R  81.6  0.1   6:30.16 avconv 

Despite encoding 14 different video files at once, the cpu usage hovers at around only 50-55%. If I am running less streams at once, the individual streams will run up to about 400 %CPU, where as pictured above they only use about 80-90 %CPU.

This is the only thing I want this box to do, is it possible to make more use of the CPU power or is my premise flawed?

I have thus far experimented with renice, to utterly no effect. I set NI of -20 and 19 it made no greater or less cpu usage overall or on individual encodes.

I did some reading about vm.swappiness, but I didn't get a clear grasp for my use case if more or less swappiness would be better. Could this increase performance?

More details: The machine I'm including top from is a Dell R900 running Ubuntu 14.04. cat /proc/cpuinfo lists 16 Intel(R) Xeon(R) CPU E7330 @ 2.40GHz processors and cat /proc/meminfo 16 Gb Ram. The other machines in the group are similar and show the same behavior. All of this encoding is DV video to Mpeg2 video, sometimes we encode other kinds of video too. The source for these encodes are on a networked raid that is not writing anything as of the top post above, and gets read speeds in the 300+Mb/s. The destination is also networked raids, not quite as fast, but the resulting mpeg data is much smaller than the input data.

4
  • Your system is using over 90% CPU in your top example above. 9.6 id = 9.6% idle. – yoonix May 22 '15 at 22:13
  • @yoonix wow, I feel a bit silly after staring at that much of the day... I guess my real question perhaps should be why is my "sy" ( system ) usage at 36%... I'll have to consider that and perhaps ask a new question. Thanks – markjwill May 22 '15 at 22:32
  • My guess is it's probably due to (or made worse by) running so many copies. I've never used avconv but a quick look at the documentation leads me to think the -threads option will help your original issue. If you really want to track down what the system CPU is this blog post can probably get you started. – yoonix May 22 '15 at 22:48
  • I suspect that at some point, your combined disk and network usage will be the bottleneck. Try removing a couple of task and check if the CPU usage diminishes dramatically. – jcoppens May 23 '15 at 0:03
0

First a high end GPU will boost encoding by orders of magnitude. Usually a high end video card will reduce rendering from hours to minutes. You may need a NVIDIA GPU.

‘-hwaccel_device[:stream_specifier] hwaccel_device (input,per-stream)’

    Select a device to use for hardware acceleration.

    This option only makes sense when the ‘-hwaccel’ option is also specified. Its exact meaning depends on the specific hardware acceleration method chosen.
    ‘vdpau’

        For VDPAU, this option specifies the X11 display/screen to use. If this option is not specified, the value of the DISPLAY environment variable is used
    ‘dxva2’

        For DXVA2, this option should contain the number of the display adapter to use. If this option is not specified, the default adapter is used. 

In addition you need this for the CPU

‘-threads[:stream_specifier] integer (input/output,video)’

    Possible values:

    ‘auto’

        autodetect a suitable number of threads to use 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.