I have a set of 4-5 machines which almost exclusively encode video all day, and sometimes all night too. I've searched many times to find ways to optimize them for this purpose, but not had any success.
My premise: I should be able to use nearly all of my cpu for video encoding, if and only if, I am running multiple encoding processes simultaneously, which I am.
The Problem: ( from 'top' )
Tasks: 382 total, 2 running, 380 sleeping, 0 stopped, 0 zombie %Cpu(s): 51.6 us, 36.8 sy, 0.0 ni, 9.6 id, 0.0 wa, 2.0 hi, 0.0 si, 0.0 st KiB Mem: 16425680 total, 16171664 used, 254016 free, 30492 buffers KiB Swap: 16769020 total, 406056 used, 16362964 free. 15198692 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31308 root 20 0 486568 26000 5560 S 92.2 0.2 188:10.72 avconv 10340 root 20 0 475832 19920 5532 S 91.2 0.1 13:40.42 avconv 12547 root 20 0 473832 23672 5528 S 89.9 0.1 5:40.23 avconv 8179 root 20 0 477132 21200 5560 S 89.2 0.1 71:50.15 avconv 9059 root 20 0 472776 17960 5540 S 89.2 0.1 48:12.62 avconv 9798 root 20 0 477812 16976 5532 S 88.9 0.1 25:23.15 avconv 8725 root 20 0 476068 21696 5560 S 87.2 0.1 62:12.02 avconv 12657 root 20 0 472332 22012 5524 S 86.9 0.1 6:21.28 avconv 12915 root 20 0 473732 41888 5516 S 86.6 0.3 5:22.63 avconv 8578 root 20 0 476840 24100 5536 S 85.6 0.1 63:17.07 avconv 7822 root 20 0 472740 18940 5536 S 85.2 0.1 75:22.04 avconv 9948 root 20 0 485420 18116 5560 S 84.9 0.1 22:18.23 avconv 7356 root 20 0 476744 20084 5560 S 84.6 0.1 92:31.91 avconv 9425 root 20 0 471096 19684 5552 S 82.9 0.1 37:48.28 avconv 12464 root 20 0 475004 24304 5532 R 81.6 0.1 6:30.16 avconv
Despite encoding 14 different video files at once, the cpu usage hovers at around only 50-55%. If I am running less streams at once, the individual streams will run up to about 400 %CPU, where as pictured above they only use about 80-90 %CPU.
This is the only thing I want this box to do, is it possible to make more use of the CPU power or is my premise flawed?
I have thus far experimented with renice, to utterly no effect. I set NI of -20 and 19 it made no greater or less cpu usage overall or on individual encodes.
I did some reading about vm.swappiness, but I didn't get a clear grasp for my use case if more or less swappiness would be better. Could this increase performance?
More details: The machine I'm including top from is a Dell R900 running Ubuntu 14.04. cat /proc/cpuinfo
lists 16 Intel(R) Xeon(R) CPU E7330 @ 2.40GHz processors and cat /proc/meminfo
16 Gb Ram. The other machines in the group are similar and show the same behavior. All of this encoding is DV video to Mpeg2 video, sometimes we encode other kinds of video too. The source for these encodes are on a networked raid that is not writing anything as of the top post above, and gets read speeds in the 300+Mb/s. The destination is also networked raids, not quite as fast, but the resulting mpeg data is much smaller than the input data.
9.6 id
= 9.6% idle.-threads
option will help your original issue. If you really want to track down what the system CPU is this blog post can probably get you started.