I'm in the process of setting up what I hope will be a robust and scalable video processing cluster for a website I'm creating.
Currently, I am processing video uploads with FFMPEG on the same web server on which my site is running.
This works well for now, but I anticipate this single server would soon be unable to cope with the amount of video processing that needs to be done, resulting in a backlog of ever-increasing length.
I would prefer to have a scalable solution, in which I could create a video processing cluster, of which individual jobs would be doled out between the nodes.
I would also like the ability to be able to add/remove and enable/disable individual processing nodes relatively painlessly - so that we scale up and down to suit our needs or financial concerns.
Our target webserver for the core site is running CentOS 6 (our current single-seat development server is running Ubuntu 12.04 LTS, which is our preferred environment for FFMPEG conversion). Our website is utilising nothing special, PHP 5.3 and above, MySQL 5, Apache 2.2.xxx.
How would one go about setting up a video processing cluster of Ubuntu VPS's, not so much concerned with the DNS/Domain Management aspects of this, but rather setting up and deploying jobs to the individual nodes?
When I was setting up my single-seat environment 6 weeks ago, I stumbled across GearMan. Does this look able to perform the task, or does anybody who might have used it have any useful notes/suggestions?
Do any of you guys have some knowledge that could help me here?