Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Does anyone know of a current, active solution to encoding x264 videos across many computers (via the network) to increase encoding FPS?

Brownie points for cross-platform and open source, but just so you all know, I usually use Windows.


Programs that I have heard of, and why I do not believe they are suitable:

  • x264farm: Not actively developed. Good interface, but does not support two-pass encoding, and fails with newer x264 builds.
  • ELDER: Again, not actively developed, but my issue was that it didn't work with new x264 builds, and it was very difficult to configure (read: randomly stopped working).

While I don't absolutely need a program which is being actively developed, I would like one that supports two-pass encoding, and works with new(er) x264 builds.


Additional information: So far, I've offered (and awarded!) two separate bounties on this question since I first posted it over two years ago, and I still haven't found a solution to this problem. What I'm looking for basically is a simple program to allow me to encode x264 videos using the processing power of multiple computers connected over a LAN.

Furthermore, it would be nice if it worked with new(er) x264 builds, and supported two-pass encoding. If at any time someone has an updated answer, or a new solution to this problem, please post it and it will be given some consideration.

share|improve this question
    
Still working on this. x264farm is just the render manager, it seems like you should be able to place any version of x264 you like on the slave pc's. Have you tried this, and what errors pop up if you did? –  Keck Sep 3 '09 at 14:51
1  
1  
I realize this is an old thread, but I think I should share my personal experience. Don't distribute one job to multiple machines, it's a waste of time, distributing to multiple cores already decreases performance, and there's multiple physical processor, then multiple machines, each with IO problem and latency. That being said, use it when only really needed, if there's multiple files (jobs), distribute by file, I believe Squeeze can distribute load across several machines, but that's quite expensive. –  Shane Hsu Dec 10 '13 at 14:13
    
@ShaneHsu thank you for sharing. I first wrote this question over four years ago, and at that time, the machine I was using to do this work was not nearly as powerful as the one I have now, so it made a lot more sense back then to go this route. Today, I'd have to agree with you - if render speed becomes an issue, it's best to offload the entire job to another machine, rather than split a single job into multiple chunks (and let one h.264 encoder instance take care of any multithreaded/multicore encoding if necessary). –  Breakthrough Dec 10 '13 at 15:13

10 Answers 10

up vote 6 down vote accepted

You could render separate chunks of the video, and use VirtualDub to stitch it all together with its Copy mode (where it does no encoding). It's not real distributed encoding or anything, but simplest solutions sometimes work the best.

share|improve this answer
3  
Again, the only problem with this is that there will be a quality loss, due to the placement of I/B frames when rendering the video. A scene detection algorithm would need to be used to determine where to split it, and somehow, you would need to split the video at exactly that frame... –  Breakthrough Sep 7 '09 at 15:57
    
VirtualDub does have those "green-and-red" icons that should serve in scene switch detection. If my memory from a few years ago serves me correctly, it worked quite nicely. But then again, I'm an amateur when it comes to video and video encoding. –  Ivan Vučica Sep 12 '09 at 0:25
    
AFAIK VirtualDub has a "go to next frame" command. I'd just split it manually. –  Camilo Martin Dec 8 '11 at 18:55

You could also try using this , its a parallel/distributed encoding software for windows and works well and scales nicely too.

Try googling for xcode Parallel encoder.

These links should provide more information.

http://superscalar.pbworks.com/

share|improve this answer
    
Unrelated: The naming looks ripped right off Apple's Xcode document about how parallel compilation worked with Xgrid. (A IDE versus a video encoder) –  Chealion Aug 10 '10 at 19:33
    
ic, i'm not a mac user but you should try this, works only on windows though. I have a setup with around 10 Ghz of combined processing power and a 90 min long video takes on an average 30-32 mins for conversion ( x.264 / AAC / 1800 kbs vbr/256 kbs audio ). –  dxblitzx Aug 10 '10 at 20:37
    
Thank you for your response. I have changed this to the current correct answer, since this solution is the closest to what I was looking for! :) –  Breakthrough Sep 9 '10 at 15:28

While it might be a bit of a overkill suggestion, Rhozet Carbon Server can pull together multiple Carbon Coder instances for the work you've described.

Website for Rhozet Carbon Server

Multiple Carbon Coder nodes can be configured as a transcoding farm, controlled by one or more Carbon Servers. Carbon Server allows for automated processing of high-volume transcoding tasks, server-controlled failover of Carbon Coder nodes, as well as managing job distribution, job prioritization, load balancing, FTP transfer, status monitoring, and job notification.

share|improve this answer

You might have a look at Media Encoding Cluster :

Media Encoding Cluster is the first Open Source Cluster Encoding Solution that is written in C/C++ for distributed Media(Video and Audio) Encoding.

Media Encoding Cluster is an extensible video encoder, which uses a lightweight peer-to-peer grid to leverage the processing power of regular PCs for the purpose of distributing the encoding of highly compressed video, for example MPEG4 and H.264

It distributes Video Chunks over the Network to Client Nodes and parallelize the Encoding Task for one File over even more than one Computer to reduce the Encoding Time per File.

Another approach is offered for Nvidia by Badaboom ($39.99 with trial), also reviewed here :

Elemental's Badaboom uses Nvidia's CUDA interface to do lots of the grunt work of DVD ripping by using the GPU instead of your musty old CPU.

In the same way, there is also Avivo Video Converter for ATI Radeon, described in wikipedia, although it might take some doing to get it working.

share|improve this answer
    
@Breakthrough: Have you looked at these products ? –  harrymc Mar 18 '11 at 19:41

the simple fact is NON of the world's Developer's has to date bothered to write and submit distributed TCP:IP/UDP generic encoding client/server patches for a current x264, as of today thats 1745 see x264.nl/

the generic client/server model is well understood, as is the clean x264 code-base, and asking for clarification of any x264 code is a simple matter of joining x264 dev IRC channel and asking, within minutes you will usually have a key x264 Dev or two answer your query in how that code section works, and even get practical idea's of how you might re-write your evolving code to better fit the x264 (and x262 a new Mpeg2 encoder based on the x264 world class framework being worked on right now) model.

So if Your a Developer then the very best thing you could do for the future of quality and profession 32/64 bit x264 distributed video encoding is actually write these required basic client/server patch's to make one instance of x264 or a seperate web/GUI app interface with this new client/server x264 API code you write, to actively look for, and assign and pass on the fly separate encode sections of a single video to any new matching managed x264 client code you also write.

your new clients/server truly distributed encode base patches dont even need to be the greatest, just basic but working and fully functioning C code that gets tested and used doom10.org/index.php?action=unread

, as theres one thing the x264 dev's seem to love to do , and thats take the existing slow C code and write optimized versions of it, section by section , but you need to actually submit the (patches welcome) actual beta code first against the latest branch OC

it's got to be worth looking into, and actually making the effort to code these x264 server to many x264 clients patch's today as x264 just got 10bit depth encoding capability's (that means high quality High, High 10, High 4:2:2 H.264 compute intensive profiles are now available to everyone for free with x264) added.

to be optimized for extra speed with assembly very soon http://mailman.videolan.org/pipermail/x264-devel/2010-October/007858.html

but even a single 8 core machine will struggle to provide highest quality output in a reasonable time with 1080P, and soon 2K and 4K super high Def etc, a real easy to set up and use distributed x264/H.264 native encode option is only a patch or two away So.

if your a dev ,PLEASE dont wait, do it today.

share|improve this answer
    
Actually, I thought about doing this. The major problem is not actually getting two computers to perform the calculations, but rather transferring the working-set data between machines. It's a lot easier to move data in and out of RAM on a single machine (in however many gigabytes per second), but much slower on a LAN (maxing out at 100 megabytes per second). –  Breakthrough Jan 4 '11 at 12:43

I'm a BIG fan of Sony Vegas for Windows video editing... and there's a feature called Network Render. :) Yums.

Sony Vegas Workflow

EDIT : Not too sure if this is a viable solution, but instead of trying to find a video-encoding application that supports network render, I tried to find a software that enables any application to take advantage of distributed computing. And I found this - IAIDataShareServer.

It looks pretty powerful, and the sample posted results are really great. If you are going to try it, let us know how it works?

EDIT2 : IAIDataShareServer seems to be just instructing machines to run individual tasks. To that extent, I have tried to source for other distributed computing solutions, and list out a few promising ones.

  1. JPPF
  2. XOREAX
  3. DCEZ (This one looks good)
share|improve this answer
2  
You sure about that? forums.creativecow.net/thread/24/895788 –  Breakthrough Sep 2 '09 at 19:18
2  
Thanks to you, now I'm definitely negative on that. ;) –  caliban Sep 2 '09 at 19:45
1  
@Breakthrough : hey mate, new possible solution found. Untested by myself though. See edited answer. Good luck! –  caliban Sep 7 '09 at 16:05
1  
@scopedreams: I saw that, and instantly thought it was perfect... Unfortunately, that distributed data share just runs instances of programs on each computer connected to it - useful for running many jobs, with each client tackling a single job at a time... But in my case, I want just one job to be computed in parallel amongst many computers. –  Breakthrough Sep 7 '09 at 19:55
1  
@Breakthrough : argh darn, back to trawling the web i guess. –  caliban Sep 7 '09 at 19:58

You could try with mythtv which could stream from the web app (basically you treat the videos as saved recordings) to many windows boxes using the windows mythtv app or VLC

share|improve this answer
    
Please clarify how the hell MythTV is remotely able to achieve what the O.P. is asking for. –  Jeremy Visser Apr 3 '11 at 9:35
    
MythTV with plugins can do 'broadcasts' to computers with encoding. my old school use to show live TV over it to all classrooms at once, via a tv tuner, and about a 30 second delay from live TV. Not exactly sure on the details of how it was done, but it didn't require extra software. –  alpha1 Jul 9 '11 at 3:14
    
I'm sorry, but that is still not remotely related to the original question. The question is about distributed encoding, not about the distribution of the encoded video. –  Jeremy Visser Jul 14 '11 at 9:09
    
-1 because you didn't read the question. –  Camilo Martin Dec 8 '11 at 19:01

For Mac OS X 10.5 (I am not sure of compatibility for 10.6) there used to be VisualHub, which would allow you to set up a grid farm on your local network. Now it's discontinued and ReduxEncoder showed up as it's replacement, but i can't seem to find the options for that.

share|improve this answer

It's beta, but functional. It's not quite as straightforward, but it works. It IS windows based and free.

ELDER from some Doom9 guys

share|improve this answer
1  
I saw that as well, but was hoping for something comparable to x264farm - there is no quality hit with x264farm... Also, the project has been abandoned for quite some time. –  Breakthrough Sep 2 '09 at 19:20
    
I originally awarded a 50 point bounty to this answer, because it was the closest solution at that time. However, this program did have some quality loss as compared to a single-computer encoder. I am hoping to avoid the hit on quality. –  Breakthrough Mar 12 '11 at 14:35
    
@Breakthrough What if you aim a bit higher, like if it makes it 10% worse make the settings (detail/framesize/etc) 10% higher? –  tobylane Apr 10 '11 at 16:18
    
@tobylane, the issue is with the placement of I/B frames when rendering the video. A scene detection algorithm would need to be used to determine where to split it, and somehow, you would need to split the video at exactly that frame. Depending on the source material, this is often impossible to do perfectly, and thus encoding a whole video at once will usually yield better quality then rendering it in chunks. –  Breakthrough Apr 10 '11 at 16:33
    
@Breakthrough x264 by default has a maximum GOP of 250 frames, with HD material even less. It WILL close the GOP sometime (unless you tweak it not to), and then there will be no quality loss if you slit right where a GOP would end, unfortunately that's not very predictable. In any case, in a 1.5-hour long movie, splitting it into 6 15-min. chunks right at scene changes wouldn't hurt much compressability. And it does help! –  Camilo Martin Dec 8 '11 at 19:00

For users of Final Cut Studio (Mac only), the x264 QuickTime component works remarkably well when used with cluster created using QMaster. Load your movie into Compressor and away it goes. In tests I found decent speed increases especially when working on a shared storage point.

share|improve this answer
3  
Damn... I'm a Windows user. That looks pretty cool though, and similar to what I'm looking for - I just wish it was multi-platform! –  Breakthrough Aug 21 '09 at 22:06

protected by studiohack May 3 '11 at 9:38

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.