Issue
I want to stream raw RGB24 frames to ffmpeg stdin, and pipe h.264 from ffmpeg into VLC, which then serves an HTTP MJPEG stream on some arbitrary port. I can get results with either program independently but I can't get the piping to work. I doubt it matters, but this is a Raspberry Pi 4B rev 1.2 with a v1 5MP camera running Pi OS ARM32 (whatever version was newest about July 2020).
Details
What I have working today is a .NET Core program using MMAL with a camera on a headless Raspberry Pi which successfully outputs 640x480 24FPS h.264 to VLC stdin, and VLC is able to relay this as an MJPEG stream over HTTP (with line breaks for readability):
cvlc stream:///dev/stdin
--sout "#transcode{vcodec=mjpg,vb=2500,fps=24,acodec=none}
:standard{access=http{mime=multipart/x-mixed-replace;
boundary=7b3cc56e5f51db803f790dad720ed50a},mux=mpjpeg,
dst=:8554/}" :demux=h264
This ends up being accessible via browser across my LAN at http://raspberrypi.local:8554/.
However, the application needs to post-process each video frame. There is no reasonable way to run the processed frames through the hardware h.264 encoder because of the way the MMAL processing pipeline works. As a result, I can only output raw RGB24 frames. (Before anyone points out I can intercept MMAL buffers in places like ports and connections, at higher resolutions a frame spans multiple buffers and the type of processing I need to do requires a full frame.)
I'm also familiar with sending to ffmpeg via stdin. I am trying to use ffmpeg to convert the raw frames to h.264 on the fly, then pipe that into VLC, but I can't seem to find a combination that works. (According to the muxer matrix on VLC's streaming features page, VLC doesn't support raw input to MJPEG output.)
To simplify initial testing, I output the raw RGB24 frames to a file. I can get ffmpeg to generate a valid h.264 file with the following settings (including some options like ultrafast that I'll want for streaming, not strictly necessary for file output), so I think this configuration is correct (again with line breaks):
ffmpeg -f rawvideo -c:v rawvideo -pix_fmt rgb24 -s:v 640x480 -r 24
-i /media/ramdisk/input.raw
-f h264 -c:v libx264 -preset ultrafast -tune zerolatency -framerate 24
/media/ramdisk/output.h264
It seems to me that I should be able to replace the last output file argument with - then pipe to that same VLC command, but it does not work. Ultimately, these two commands will end up in a .sh script so that I can launch it with a single .NET Process.
Console Output
The console output (with ffmpeg -hide_banner to suppress output of the switch settings) starts with some VLC messages, which is normal:
VLC media player 3.0.11 Vetinari (revision 3.0.11-0-gdc0c5ced72)
[00663e18] vlcpulse audio output error: PulseAudio server connection failure: Connection refused
[00693e58] main interface error: no suitable interface module
[005e8b58] main libvlc error: interface "globalhotkeys,none" initialization failed
[00693e58] dummy interface: using the dummy interface module...
Followed by some ffmpeg output:
[rawvideo @ 0x9672c0] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from '/media/ramdisk/input.raw':
Duration: 00:00:14.96, start: 0.000000, bitrate: 176947 kb/s
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 640x480, 176947 kb/s, 24 tbr, 24 tbn, 24 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x973520] using cpu capabilities: ARMv6 NEON
[libx264 @ 0x973520] profile High 4:4:4 Predictive, level 3.0, 4:4:4 8-bit
Output #0, h264, to 'pipe:':
Metadata:
encoder : Lavf58.20.100
Stream #0:0: Video: h264 (libx264), yuv444p, 640x480, q=-1--1, 24 fps, 24 tbn, 24 tbc
Metadata:
encoder : Lavc58.35.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Then more VLC output -- the first two lines are unique to this setup, it isn't what I see when I output h.264 directly to VLC stdin:
[b2705a68] main stream error: unknown query 0x30e in demux_vaControlHelper
[b0b00e18] mmal_codec generic: VCSM init succeeded: CMA
[b0b01110] avcodec encoder error: Unknown option "rc_buffer_aggressivity"
Then ffmpeg starts outputting:
frame= 359 fps= 29 q=22.0 Lsize= 754kB time=00:00:14.95 bitrate= 413.0kbits/s speed= 1.2x
video:754kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
[libx264 @ 0x973520] frame I:2 Avg QP:19.50 size: 18098
[libx264 @ 0x973520] frame P:357 Avg QP:22.00 size: 2062
[libx264 @ 0x973520] mb I I16..4: 100.0% 0.0% 0.0%
[libx264 @ 0x973520] mb P I16..4: 11.7% 0.0% 0.0% P16..4: 22.5% 0.0% 0.0% 0.0% 0.0% skip:65.8%
[libx264 @ 0x973520] coded y,u,v intra: 5.1% 0.3% 0.2% inter: 7.6% 1.1% 0.8%
[libx264 @ 0x973520] i16 v,h,dc,p: 33% 29% 22% 16%
[libx264 @ 0x973520] kb/s:413.00
Nothing else happens. Attempting to connect over HTTP never works at any point during this processing. I've spent hours searching and trying many combinations, and the commands listed here are the only ones that don't result in some kind of error.
Note I'm not looking for answers that involve additional applications or protocols. I'm aware there are many ways to accomplish streaming output, but I have reasons I wish to use ffmpeg, VLC, and MJPEG only. I'm not tied to h.264 between ffmpeg and VLC if there is a more efficient way to accomplish the same thing.
