I currently have two ffmpeg videos. Video one is a matte key file , that looks like so

Example of output

Video two, would be the normal video without the background and just the person.

How would I go about merging these two videos in ffmpeg to have it look like this

enter image description here

I've tried the command

ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[0:v][1:v]alphamerge" -shortest -c:v qtrle -an output.mp4

but I get the following error

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'word.mp4':
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.24, start: 0.000000, bitrate: 741 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 738 kb/s, 25.79 fps, 25.79 tbr, 11040 tbn, 51.59 tbc (default)
      handler_name    : VideoHandler
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'word.matte.mp4':
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.19, start: 0.000000, bitrate: 493 kb/s
    Stream #1:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 568x320 [SAR 1:1 DAR 71:40], 491 kb/s, 26 fps, 26 tbr, 13312 tbn, 26 tbc (default)
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 (h264) -> alphamerge:main
  Stream #1:0 (mpeg4) -> alphamerge:alpha
  alphamerge -> Stream #0:0 (qtrle)
Press [q] to stop, [?] for help
[swscaler @ 0x5619d30b9900] No accelerated colorspace conversion found from yuv420p to argb.
[Parsed_alphamerge_0 @ 0x5619d2f57700] Input frame sizes do not match (1280x720 vs 568x320).
[Parsed_alphamerge_0 @ 0x5619d2f57700] Failed to configure output pad on Parsed_alphamerge_0
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0

Any help would be appreciated. Here is a link to both video files, the matte file, and the normal video

Edit: Here is the example of this working. https://backgroundremover.app/video


Make a new mask to match the video size

Your mask is smaller than the video (568x320 vs 1280x720). Optimal solution is to make a new mask that matches the size and aspect ratio of the video. This will avoid the need to upscale the mask and will provide the best results. Avoid upscaling when possible.

If you have to use the original mask

You can use the scale2ref filter to upscale the mask to match the video:

ffmpeg -i video.mp4 -i mask.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge=shortest=1" -c:v qtrle output.mov

See the scale2ref filter documentation for more details.

  • LIterally what I said.
    – WG481
    May 10 at 17:27
  • @llogan I have updated the question with the example video files. Your command does not give an error, but does not remove the background and instead just gives the same video with the background
    – nadermx
    May 10 at 18:22
  • @llogan I have gone ahead and put the originals, with the different sizes. Also is there any way to get it to auto detect the range, although that may be outside the scope of this question. And I currently need to upscale the matte file, as to create a original size matte is too expensive in terms of cpu/gpu for my program currently.
    – nadermx
    May 11 at 17:46
  • @nadermx See updated answer showing basic scale2ref usage.
    – llogan
    May 11 at 18:03
  • 1
    @nadermx Blame VLC. Try mpv. Verify that the mask works by outputting PNG: ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge" output_%03d.png or overlaying on color ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge1[fg];color=c=magenta:s=1280x720[bg];[bg][fg]overlay=format=auto:shortest=1,format=yuv420p" output.mp4
    – llogan
    May 12 at 16:20

I read through your error log and found this interesting line of text:
[Parsed_alphamerge_0 @ 0x5619d2f57700] Input frame sizes do not match (1280x720 vs 568x320).

In this error line, it says the sizes of both frames don't match, and thus, cannot be put together. By downscaling/upscaling one of your files to match the other's resolution, the videos should be put together with ease.

  • I agree with llogan that your answer simply showed me the error log, which I had already read, but was unsure of how to fix.
    – nadermx
    May 12 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.