The problem is in the VirtualBox distribution itself, which now lacks the required modules.
If you took a look in the module VBoxHeadless.cpp, you would see that for this to function, the following conditions must be fulfilled :
- VBOX_FFMPEG must be defined
- The VBoxFFmpegFB shared library must be available with entry-point VBoxRegisterFFmpegFB
In the thread VBoxHeadless "Failed to load the video capture extension" from 2009, someone named xantus has tried to create the required conditions by modifying the sources accordingly, but this didn't work.
I suggest that the proper way to go about it, is to address the developers on the VirtualBox forums and convince them to get it working again, although this option seems to have been abandoned since at least 2009.