31

Does anyone know of a current, active solution to encoding x264 videos across many computers (via the network) to increase encoding FPS?

Brownie points for cross-platform and open source, but just so you all know, I usually use Windows.


Programs that I have heard of, and why I do not believe they are suitable:

  • x264farm: Not actively developed. Good interface, but does not support two-pass encoding, and fails with newer x264 builds.
  • ELDER: Again, not actively developed, but my issue was that it didn't work with new x264 builds, and it was very difficult to configure (read: randomly stopped working).

While I don't absolutely need a program which is being actively developed, I would like one that supports two-pass encoding, and works with new(er) x264 builds.


Additional information: So far, I've offered (and awarded!) two separate bounties on this question since I first posted it over two years ago, and I still haven't found a solution to this problem. What I'm looking for basically is a simple program to allow me to encode x264 videos using the processing power of multiple computers connected over a LAN. Furthermore, it would be nice if it worked with new(er) x264 builds, and supported two-pass encoding.

If at any time someone has an updated answer, or a new solution to this problem, please post it and it will be given some consideration.


2016 Update:

After much of my work experience with computer/machine vision, I've come to realize that the overhead associated with the large amount of shared data/memory, and the potential bottleneck it presents, might outweigh the potential benefits.

While I would still love to find something that would allow me to harness the idle computing power of several devices, for now, modern GPGPU-based encoders are a much better approach if you need improved/real-time encoding. This is what most cloud-based video encoding platforms provide (which another alternative if you're into SaaS or cloud computing), albeit on a larger scale.

6
  • Still working on this. x264farm is just the render manager, it seems like you should be able to place any version of x264 you like on the slave pc's. Have you tried this, and what errors pop up if you did?
    – Keck
    Sep 3, 2009 at 14:51
  • 1
  • 1
    I realize this is an old thread, but I think I should share my personal experience. Don't distribute one job to multiple machines, it's a waste of time, distributing to multiple cores already decreases performance, and there's multiple physical processor, then multiple machines, each with IO problem and latency. That being said, use it when only really needed, if there's multiple files (jobs), distribute by file, I believe Squeeze can distribute load across several machines, but that's quite expensive.
    – Shane Hsu
    Dec 10, 2013 at 14:13
  • @ShaneHsu thank you for sharing. I first wrote this question over four years ago, and at that time, the machine I was using to do this work was not nearly as powerful as the one I have now, so it made a lot more sense back then to go this route. Today, I'd have to agree with you - if render speed becomes an issue, it's best to offload the entire job to another machine, rather than split a single job into multiple chunks (and let one h.264 encoder instance take care of any multithreaded/multicore encoding if necessary). Dec 10, 2013 at 15:13
  • I'm looking to do the same thing, but unfortunately it looks like this thread is mostly filled with half-baked solutions or projects that don't exist anymore. While it looks like your need to do this has faded, if you have any more information on possible solutions since the last time you updated this please let me know.
    – Locksleyu
    Mar 16, 2016 at 17:02

9 Answers 9

6

You could render separate chunks of the video, and use VirtualDub to stitch it all together with its Copy mode (where it does no encoding). It's not real distributed encoding or anything, but simplest solutions sometimes work the best.

5
  • 5
    Again, the only problem with this is that there will be a quality loss, due to the placement of I/B frames when rendering the video. A scene detection algorithm would need to be used to determine where to split it, and somehow, you would need to split the video at exactly that frame... Sep 7, 2009 at 15:57
  • VirtualDub does have those "green-and-red" icons that should serve in scene switch detection. If my memory from a few years ago serves me correctly, it worked quite nicely. But then again, I'm an amateur when it comes to video and video encoding. Sep 12, 2009 at 0:25
  • AFAIK VirtualDub has a "go to next frame" command. I'd just split it manually. Dec 8, 2011 at 18:55
  • @Breakthrough So all you need is a filter that splits a video input into chunks at scene change boundaries (so that these can then be encoded separately)? That's simple enough. Is there some other issue? Mar 29, 2015 at 8:03
  • @GroovyDotCom well in addition to that, all the supporting software (e.g. a server to initiate the splitting filter, distribute it to all the client nodes running encoders, queue the jobs, transfer the files back to the main server, and re-merge the result) still needs to be dealt with, and this still doesn't address any potential quality/efficiency issues with the method of encoding a large video in individual segments. Also note that this question is nearly six years old at this point, so I'm sure a lot has changed since then too with respect to distributed encoding. Mar 31, 2015 at 9:02
4
+50

It's beta, but functional. It's not quite as straightforward, but it works. It IS windows based and free.

ELDER from some Doom9 guys

5
  • 2
    I saw that as well, but was hoping for something comparable to x264farm - there is no quality hit with x264farm... Also, the project has been abandoned for quite some time. Sep 2, 2009 at 19:20
  • 1
    I originally awarded a 50 point bounty to this answer, because it was the closest solution at that time. However, this program did have some quality loss as compared to a single-computer encoder. I am hoping to avoid the hit on quality. Mar 12, 2011 at 14:35
  • @Breakthrough What if you aim a bit higher, like if it makes it 10% worse make the settings (detail/framesize/etc) 10% higher?
    – tobylane
    Apr 10, 2011 at 16:18
  • @tobylane, the issue is with the placement of I/B frames when rendering the video. A scene detection algorithm would need to be used to determine where to split it, and somehow, you would need to split the video at exactly that frame. Depending on the source material, this is often impossible to do perfectly, and thus encoding a whole video at once will usually yield better quality then rendering it in chunks. Apr 10, 2011 at 16:33
  • 2
    @Breakthrough x264 by default has a maximum GOP of 250 frames, with HD material even less. It WILL close the GOP sometime (unless you tweak it not to), and then there will be no quality loss if you slit right where a GOP would end, unfortunately that's not very predictable. In any case, in a 1.5-hour long movie, splitting it into 6 15-min. chunks right at scene changes wouldn't hurt much compressability. And it does help! Dec 8, 2011 at 19:00
3

You could also try using this , its a parallel/distributed encoding software for windows and works well and scales nicely too.

Try googling for xcode Parallel encoder.

These links should provide more information.

http://superscalar.pbworks.com/

3
  • Unrelated: The naming looks ripped right off Apple's Xcode document about how parallel compilation worked with Xgrid. (A IDE versus a video encoder)
    – Chealion
    Aug 10, 2010 at 19:33
  • ic, i'm not a mac user but you should try this, works only on windows though. I have a setup with around 10 Ghz of combined processing power and a 90 min long video takes on an average 30-32 mins for conversion ( x.264 / AAC / 1800 kbs vbr/256 kbs audio ).
    – dxblitzx
    Aug 10, 2010 at 20:37
  • Thank you for your response. I have changed this to the current correct answer, since this solution is the closest to what I was looking for! :) Sep 9, 2010 at 15:28
2

For users of Final Cut Studio (Mac only), the x264 QuickTime component works remarkably well when used with cluster created using QMaster. Load your movie into Compressor and away it goes. In tests I found decent speed increases especially when working on a shared storage point.

1
  • 3
    Damn... I'm a Windows user. That looks pretty cool though, and similar to what I'm looking for - I just wish it was multi-platform! Aug 21, 2009 at 22:06
2

For Mac OS X 10.5 (I am not sure of compatibility for 10.6) there used to be VisualHub, which would allow you to set up a grid farm on your local network. Now it's discontinued and ReduxEncoder showed up as it's replacement, but i can't seem to find the options for that.

2

I'm a BIG fan of Sony Vegas for Windows video editing... and there's a feature called Network Render. :) Yums.

Sony Vegas Workflow

EDIT : Not too sure if this is a viable solution, but instead of trying to find a video-encoding application that supports network render, I tried to find a software that enables any application to take advantage of distributed computing. And I found this - IAIDataShareServer.

It looks pretty powerful, and the sample posted results are really great. If you are going to try it, let us know how it works?

EDIT2 : IAIDataShareServer seems to be just instructing machines to run individual tasks. To that extent, I have tried to source for other distributed computing solutions, and list out a few promising ones.

  1. JPPF
  2. XOREAX
  3. DCEZ (This one looks good)
6
  • 3
    You sure about that? forums.creativecow.net/thread/24/895788 Sep 2, 2009 at 19:18
  • 1
    @Breakthrough : hey mate, new possible solution found. Untested by myself though. See edited answer. Good luck!
    – caliban
    Sep 7, 2009 at 16:05
  • 2
    @scopedreams: I saw that, and instantly thought it was perfect... Unfortunately, that distributed data share just runs instances of programs on each computer connected to it - useful for running many jobs, with each client tackling a single job at a time... But in my case, I want just one job to be computed in parallel amongst many computers. Sep 7, 2009 at 19:55
  • 1
    @Breakthrough : argh darn, back to trawling the web i guess.
    – caliban
    Sep 7, 2009 at 19:58
  • 1
    @Breakthrough : Updated my answer to provide a list of distributed computing clients. Again, untested. Don't worry about accepting my answer, i'm doing it to learn something new for myself too. :)
    – caliban
    Sep 7, 2009 at 20:04
1

the simple fact is NON of the world's Developer's has to date bothered to write and submit distributed TCP:IP/UDP generic encoding client/server patches for a current x264, as of today thats 1745 see x264.nl/

the generic client/server model is well understood, as is the clean x264 code-base, and asking for clarification of any x264 code is a simple matter of joining x264 dev IRC channel and asking, within minutes you will usually have a key x264 Dev or two answer your query in how that code section works, and even get practical idea's of how you might re-write your evolving code to better fit the x264 (and x262 a new Mpeg2 encoder based on the x264 world class framework being worked on right now) model.

So if Your a Developer then the very best thing you could do for the future of quality and profession 32/64 bit x264 distributed video encoding is actually write these required basic client/server patch's to make one instance of x264 or a seperate web/GUI app interface with this new client/server x264 API code you write, to actively look for, and assign and pass on the fly separate encode sections of a single video to any new matching managed x264 client code you also write.

your new clients/server truly distributed encode base patches dont even need to be the greatest, just basic but working and fully functioning C code that gets tested and used doom10.org/index.php?action=unread

, as theres one thing the x264 dev's seem to love to do , and thats take the existing slow C code and write optimized versions of it, section by section , but you need to actually submit the (patches welcome) actual beta code first against the latest branch OC

it's got to be worth looking into, and actually making the effort to code these x264 server to many x264 clients patch's today as x264 just got 10bit depth encoding capability's (that means high quality High, High 10, High 4:2:2 H.264 compute intensive profiles are now available to everyone for free with x264) added.

to be optimized for extra speed with assembly very soon http://mailman.videolan.org/pipermail/x264-devel/2010-October/007858.html

but even a single 8 core machine will struggle to provide highest quality output in a reasonable time with 1080P, and soon 2K and 4K super high Def etc, a real easy to set up and use distributed x264/H.264 native encode option is only a patch or two away So.

if your a dev ,PLEASE dont wait, do it today.

1
  • Actually, I thought about doing this. The major problem is not actually getting two computers to perform the calculations, but rather transferring the working-set data between machines. It's a lot easier to move data in and out of RAM on a single machine (in however many gigabytes per second), but much slower on a LAN (maxing out at 100 megabytes per second). Jan 4, 2011 at 12:43
1

You might have a look at Media Encoding Cluster :

Media Encoding Cluster is the first Open Source Cluster Encoding Solution that is written in C/C++ for distributed Media(Video and Audio) Encoding.

Media Encoding Cluster is an extensible video encoder, which uses a lightweight peer-to-peer grid to leverage the processing power of regular PCs for the purpose of distributing the encoding of highly compressed video, for example MPEG4 and H.264

It distributes Video Chunks over the Network to Client Nodes and parallelize the Encoding Task for one File over even more than one Computer to reduce the Encoding Time per File.

Another approach is offered for Nvidia by Badaboom ($39.99 with trial), also reviewed here :

Elemental's Badaboom uses Nvidia's CUDA interface to do lots of the grunt work of DVD ripping by using the GPU instead of your musty old CPU.

In the same way, there is also Avivo Video Converter for ATI Radeon, described in wikipedia, although it might take some doing to get it working.

1
  • @Breakthrough: Have you looked at these products ?
    – harrymc
    Mar 18, 2011 at 19:41
1

While it might be a bit of a overkill suggestion, Rhozet Carbon Server can pull together multiple Carbon Coder instances for the work you've described.

Website for Rhozet Carbon Server

Multiple Carbon Coder nodes can be configured as a transcoding farm, controlled by one or more Carbon Servers. Carbon Server allows for automated processing of high-volume transcoding tasks, server-controlled failover of Carbon Coder nodes, as well as managing job distribution, job prioritization, load balancing, FTP transfer, status monitoring, and job notification.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .