Low-latency streaming using ffmpeg

Question

Recently, I'm trying to stream a remote desktop using ffmpeg. The goal is to being able to send what is shown on one of the computer's screen to another computer while being able to choose settings like the output resolution, etc.

It is a small nearly working proof of concept. When done, I'll be able to replace VNC by using ffmpeg streaming with x2x or synergy for forwarding keyboard and mouse events.

Now, I can start streaming using:

ffmpeg -f x11grab -s "1600x1200" -i ":0" \
    -f alsa -i pulse \
    -s 800x600 -b 200k -f mpegts - \
    | mplayer -cache 1024 -
#
# I have pulse audio configured so that `-i pulse` will
# The output can sent through for example netcat to another host
#

The latency depends on the encoding options. With these options, the delay between the screen update and the video update is arround 800ms.

Thing I'm trying to achieve:

• Reach a latency of 100ms.

• When mplayer is executed it complains that it cannot seek in lear streams and there is no audio. I still have the sound when I'm saving the output to a file and play it.

  EDIT: After adding -cache, messages to seek in linear streams no longer appear. Changing output format to -f mpegts makes the audio work.

• It would be great if the encoding wouldn't take 100% of one of the cpu cores (secondary goal).

After some reseach on the internet, I think that these problems are related to the codecs/options I should use. Yet I don't know the differences between the existing possibilities. Could you give me some options that would solve my problems? Also, is VLC a good alternative and if so what would be the equivalent commands to stream from one desktop to another?

Answer 1

Streaming from desktopA to desktopB is an interesting problem. If you don't need video/audio, then you might want to check into the NX protocol instead. It is much more efficient than VNC. FreeNX is a server and NoMachine and others make NX clients. There are other servers - free and commercial.

There is also work being performed on a high performance remote desktop, SPICE, capability in as a F/LOSS project. I think Redhat is taking the lead. http://spice-space.org/

I don't know if these are useful answers, but perhaps just leads for something more useful?

For streaming video, avoiding the overhead of a full desktop protocol is probably needed.

Answer 2

I recently needed to set up low-latency streaming with even stricter requirements than the original poster (OP). I devised a solution that works well enough to use as an additional monitor for my laptop, with smooth mouse and keyboard input.

On the sending side, I used the following ffmpeg command :

ffmpeg -video_size 1920x1080 -r 30 -framerate 30 -f x11grab -i :0.0+0x0 \
    -b:v 40M -maxrate 50M -bufsize 200M \
    -field_order tt -fflags nobuffer -threads 1 \
    -vcodec mpeg4 -g 100 -r 30 -bf 0 -mbd bits -flags +aic+mv4+low_delay \
    -thread_type slice -slices 1 -level 32 -strict experimental -f_strict experimental \
    -syncpoints none -f nut "tcp://10.0.0.1:1234"

The critical part of this command is the -g 100 argument, which adds time between keyframes, lowering the stream bitrate.

On the receiving side, I used ffplay (included with ffmpeg) to display the stream on a Raspberry Pi 3 :

ffplay -autoexit -flags low_delay -framedrop -strict experimental \
    -vf setpts=0 -tcp_nodelay 1 "tcp://10.0.0.1:1234\?listen"

While I realize the original question is over a decade old and the processing power available to the average user has improved significantly since then, these two commands provide a reasonable starting point for anyone seeking to set up a low-latency stream over a local network, even on low-end hardware.