Possible to use VP8 or VP9 encoding with Intel Quick Sync on FFmpeg?

Question

According to the chart on Wikipedia, later Intel CPUs support VP8 encoding with Quick Sync, and Kaby Lake even supports VP9 encoding.

Is there any FFmpeg build capable of utilizing this?

When I run ffmpeg -codecs on the latest Zeranoe builds, I see:

DEV.L. vp8    On2 VP8 (decoders: vp8 libvpx vp8_cuvid vp8_qsv ) (encoders: libvpx )

Running FFmpeg with -vcodec vp8_qsv for the output fails, as clearly there is no encoder for it.

Is this an issue with my build, or is this there no support in FFmpeg today?

Answer 1

This is now possible, but on Linux via VAAPI.

Here is a gist that shows you how to generate a working build on Ubuntu 16.04LTS.

For the QSV-based encoders, as tested with the opensource Intel Media SDK (on an Apollolake testbed), only the H.264/AVC and the HEVC encoders are available. Newer generations will typically expose more encoding capabilities.

Note that QuickSync H/W acceleration requires the presence of the media-driver package on the system. Some Linux distributions, such as Ubuntu 19.04LTS (and above) package it in their repositories.

Update: VP9 encoding is now enabled for IceLake (ICL+) hardware through QuickSync, and FFmpeg can take advantage of it.

Examples demonstrating the use of FFmpeg's vp9_vaapi and vp9_qsv encoder wrappers:

You will need to use FFmpeg directly, where you may optionally add the vp9_superframe and the vp9_raw_reorder bitstream filters in the same command line if you enable B-frames in the vp9_vaapi encoder.

Example:

ffmpeg -vaapi_device /dev/dri/renderD128 -hwaccel vaapi \
-i http://server:port -vf 'format=nv12|vaapi,hwupload' \
-c:v vp9_vaapi -global_quality 50 -bf 1 \
-bsf:v vp9_raw_reorder,vp9_superframe \
-f segment -segment_time 5 -segment_format_options movflags=+faststart output%03d.mp4

Adjust your input and output paths/urls as needed.

What this command does:

It will create 5 second long mp4 segments, via the segment muxer. See the usage of the movflags=+faststart , and how it has been passed as a format option to the underlying mp4 muxer via -segment_format_options flag above.

The segment lengths may not be exactly 5 seconds long, as each segment begins (gets cut on) (with) a keyframe.

However, I'd not recommend enabling B-frames in that encoder, as these bitstream filters have other undesired effects, such as mucking around with the encoder's rate control and triggering bugs like this one. This is not desirable in a production environment. This is why the scripts below do not have that option enabled, and instead, we define a set rate control mode directly in the encoder options.

If you need to take advantage of 1:N encoding with VAAPI, use these snippets:

1. If you need to deinterlace, call up the deinterlace_vaapi filter:

    ffmpeg -loglevel debug -threads 1 \
    -init_hw_device vaapi=va:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_device va -filter_hw_device va -hwaccel_output_format vaapi \
    -i 'http://server:port' \
    -filter_complex "[0:v]format=nv12|vaapi,hwupload,deinterlace_vaapi,split=3[n0][n1][n2]; \
    [n0]scale_vaapi=1152:648[v0]; \
    [n1]scale_vaapi=848:480[v1];
    [n2]scale_vaapi=640:360[v2]" \
    -b:v:0 2250k -maxrate:v:0 2250k -bufsize:v:0 360k -c:v:0 vp9_vaapi -g:v:0 50 -r:v:0 25 -rc_mode:v:0 2 \
    -b:v:1 1750k -maxrate:v:1 1750k -bufsize:v:1 280k -c:v:1 vp9_vaapi -g:v:1 50 -r:v:1 25 -rc_mode:v:1 2 \
    -b:v:2 1000k -maxrate:v:2 1000k -bufsize:v:2 160k -c:v:2 vp9_vaapi -g:v:2 50 -r:v:2 25 -rc_mode:v:2 2 \
    -c:a aac -b:a 128k -ar 48000 -ac 2 \
    -flags -global_header -f tee -use_fifo 1 \
    -map "[v0]" -map "[v1]" -map "[v2]" -map 0:a \
    "[select=\'v:0,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path0/output%03d.mp4| \
     [select=\'v:1,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path1/output%03d.mp4| \
     [select=\'v:2,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path2/output%03d.mp4"

2. Without deinterlacing:

    ffmpeg -loglevel debug -threads 1 \
    -init_hw_device vaapi=va:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_device va -filter_hw_device va -hwaccel_output_format vaapi \
    -i 'http://server:port' \
    -filter_complex "[0:v]format=nv12|vaapi,hwupload,split=3[n0][n1][n2]; \
    [n0]scale_vaapi=1152:648[v0]; \
    [n1]scale_vaapi=848:480[v1];
    [n2]scale_vaapi=640:360[v2]" \
    -b:v:0 2250k -maxrate:v:0 2250k -bufsize:v:0 2250k -c:v:0 vp9_vaapi -g:v:0 50 -r:v:0 25 -rc_mode:v:0 2  \
    -b:v:1 1750k -maxrate:v:1 1750k -bufsize:v:1 1750k -c:v:1 vp9_vaapi -g:v:1 50 -r:v:1 25 -rc_mode:v:1 2  \
    -b:v:2 1000k -maxrate:v:2 1000k -bufsize:v:2 1000k -c:v:2 vp9_vaapi -g:v:2 50 -r:v:2 25 -rc_mode:v:2 2  \
    -c:a aac -b:a 128k -ar 48000 -ac 2 \
    -flags -global_header -f tee -use_fifo 1 \
    -map "[v0]" -map "[v1]" -map "[v2]" -map 0:a \
    "[select=\'v:0,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path0/output%03d.mp4| \
     [select=\'v:1,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path1/output%03d.mp4| \
     [select=\'v:2,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path2/output%03d.mp4"

3. Using Intel's QuickSync (on supported platforms):

On Intel Icelake and above, you can use the vp9_qsv encoder wrapper with the following known limitations (for now):

(a). You must enable low_power mode because only the VDENC decode path is exposed by the iHD driver for now.

(b). Coding option1 and extra_data are not supported by MSDK.

(c). The IVF header will be inserted in MSDK by default, but it is not needed for FFmpeg, and remains disabled by default.

See the examples below:

1. If you need to deinterlace, call up the vpp_qsv filter:

    ffmpeg -nostdin -y -fflags +genpts -init_hw_device qsv=hw \
    -hwaccel qsv -hwaccel_output_format qsv \
    -filter_hw_device hw \
    -threads 1 -vsync 1 -async 1 -reinit_filter 1 \
    -i 'http://server:port' \
    -filter_complex "[0:v]format=nv12|qsv,hwupload=extra_hw_frames=64,vpp_qsv=deinterlace=2:async_depth=4,split[n0][n1][n2]; \
    [n0]vpp_qsv=w=1152:h=648:async_depth=4[v0]; \
    [n1]vpp_qsv=w=848:h=480:async_depth=4[v1];
    [n2]vpp_qsv=w=640:h=360:async_depth=4[v2]" \
    -b:v:0 2250k -maxrate:v:0 2250k -bufsize:v:0 360k -c:v:0 vp9_qsv -g:v:0 50 -r:v:0 25 -low_power:v:0 2 \
    -b:v:1 1750k -maxrate:v:1 1750k -bufsize:v:1 280k -c:v:1 vp9_qsv -g:v:1 50 -r:v:1 25 -low_power:v:1 2 \
    -b:v:2 1000k -maxrate:v:2 1000k -bufsize:v:2 160k -c:v:2 vp9_qsv -g:v:2 50 -r:v:2 25 -low_power:v:2 2 \
    -c:a aac -b:a 128k -ar 48000 -ac 2 \
    -flags -global_header -f tee -use_fifo 1 \
    -map "[v0]" -map "[v1]" -map "[v2]" -map 0:a \
    "[select=\'v:0,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path0/output%03d.mp4| \
     [select=\'v:1,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path1/output%03d.mp4| \
     [select=\'v:2,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path2/output%03d.mp4"

2. Without deinterlacing:

    ffmpeg -nostdin -y -fflags +genpts -init_hw_device qsv=hw \
    -hwaccel qsv -hwaccel_output_format qsv \
    -filter_hw_device hw \
    -threads 1 -vsync 1 -async 1 -reinit_filter 1 \
    -i 'http://server:port' \
    -filter_complex "[0:v]format=nv12|qsv,hwupload=extra_hw_frames=64,split=3[n0][n1][n2]; \
    [n0]vpp_qsv=w=1152:h=648:async_depth=4[v0]; \
    [n1]vpp_qsv=w=848:h=480:async_depth=4[v1];
    [n2]vpp_qsv=w=640:h=360:async_depth=4[v2]" \
    -b:v:0 2250k -maxrate:v:0 2250k -bufsize:v:0 2250k -c:v:0 vp9_qsv -g:v:0 50 -r:v:0 25 -low_power:v:0 2  \
    -b:v:1 1750k -maxrate:v:1 1750k -bufsize:v:1 1750k -c:v:1 vp9_qsv -g:v:1 50 -r:v:1 25 -low_power:v:1 2  \
    -b:v:2 1000k -maxrate:v:2 1000k -bufsize:v:2 1000k -c:v:2 vp9_qsv -g:v:2 50 -r:v:2 25 -low_power:v:2 2  \
    -c:a aac -b:a 128k -ar 48000 -ac 2 \
    -flags -global_header -f tee -use_fifo 1 \
    -map "[v0]" -map "[v1]" -map "[v2]" -map 0:a \
    "[select=\'v:0,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path0/output%03d.mp4| \
     [select=\'v:1,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path1/output%03d.mp4| \
     [select=\'v:2,a\':f=segment:segment_time=5:segment_format_options=movflags=+faststart]$output_path2/output%03d.mp4"

Note that we use the vpp_qsv filter with the async_depth option set to 4. This massively improves transcode performance over using scale_qsv and deinterlace_qsv. See this commit on FFmpeg's git.

Note: If you use the QuickSync path, note that MFE (Multi-Frame encoding mode) will be enabled by default if the Media SDK library on your system supports it.

References:

1. See the encoder options, including rate control methods supported:

ffmpeg -h encoder=vp9_vaapi

2. See the deinterlace_vaapi filter usage options:

ffmpeg -h filter=deinterlace_vaapi

3. On the vpp_qsv filter usage, see:

ffmpeg -h filter=vpp_qsv

For instance, if you want field-rate output rather than frame-rate output from the deinterlacer, you could pass the rate=field option to it instead:

-vf=vaapi_deinterlace=rate=field

This feature, for instance, is tied to encoders that support MBAFF. Others, such as the NVENC-based ones in FFmpeg, do not have this implemented (as of the time of writing).

Tips on maximizing performance with hardware-based encoders in FFmpeg:

Where possible, infer to the built-in docs, as with the examples shown above. They can uncover potential pitfalls that you may be able to avoid by understanding how filter chaining and encoder initialization works, unsupported features, etc, and the impact on performance.

Secondly, where possible, take advantage of 1:N transcoding where multiple outputs are required. That way, inputs and filter chain products can be re-used.

For example, you'll see that in the snippets above, we call up the deinterlacer only once, then split its' output via the split filter to separate scalers. This is done to lower the overhead that would be incurred had we called up the deinterlacer more than once, and it would've been wasteful.