3

I have a video (original.mp4), and then I have a series of PNGs with transparency that I want to overlay every frame in the video.

The strategy I'm using is:

  1. Create a video from the PNGs

    ffmpeg -framerate 60 -pattern_type glob -i images/*.png -c:v libx264 out.mp4

  2. Normalize the two videos to have the same dimensions

    ffmpeg -i out.mp4 -vf scale=1920:960,setsar=0:1,setdar=16:9 out2.mp4
    ffmpeg -i original.mp4 -vf scale=1920:960,setsar=0:1,setdar=16:9 original2.mp4

  3. Overlay the PNGs video and give it opacity

    ffmpeg -i original2.mp4 -i heatmap2.avi -filter_complex "blend=all_mode='overlay':all_opacity=0.7" result.mp4

This almost works, but the video created from the PNGs has a black background. I've also tried encoding the video as .mov and .avi, which are supposed to retain transparency. However when I did that, the black background still came back after overlaying it.

How can I seamlessly overlay the PNGs (each is a single frame of the video) as a video with transparency over the original?

[Edit 2] From Mulvya's suggestion, here is the command I tried

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i "images/*.png" -filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr]; [ovr][0:v]blend=all_mode='overlay':all_opacity=0.7[v]" -map [v] result.mp4

And the output:

ffmpeg version 2.8.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.0 (clang-700.1.76)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.8.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid --enable-vda
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'original.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    creation_time   : 2016-03-08 18:24:07
  Duration: 00:00:06.91, start: 0.023220, bitrate: 2466 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], 2331 kb/s, 29.95 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2016-03-08 18:24:07
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 102 kb/s (default)
    Metadata:
      creation_time   : 2016-03-08 18:24:07
      handler_name    : Core Media Audio
Input #1, image2, from 'images/*.png':
  Duration: 00:00:07.10, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: png, rgba(pc), 1920x960, 60 fps, 60 tbr, 60 tbn, 60 tbc
File 'result.mp4' already exists. Overwrite ? [y/N] y
[libx264 @ 0x7fa5b2000c00] using SAR=2769/2768
[libx264 @ 0x7fa5b2000c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fa5b2000c00] profile High, level 4.2
[libx264 @ 0x7fa5b2000c00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[libx264 @ 0x7fa5b2001e00] using SAR=2769/2768
[libx264 @ 0x7fa5b2001e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7fa5b2001e00] profile High, level 4.2
[libx264 @ 0x7fa5b2001e00] 264 - core 148 r2601 a0cd7d3 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'result.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], q=-1--1, 60 fps, 15360 tbn, 60 tbc
    Metadata:
      encoder         : Lavc56.60.100 libx264
    Stream #0:1: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x960 [SAR 2769:2768 DAR 2769:1384], q=-1--1, 60 fps, 15360 tbn, 60 tbc
    Metadata:
      encoder         : Lavc56.60.100 libx264
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #0:0 (h264) -> blend:bottom
  Stream #1:0 (png) -> scale2ref:default
  scale2ref:ref -> Stream #0:0 (libx264)
  blend -> Stream #0:1 (libx264)
Press [q] to stop, [?] for help
[image2 @ 0x7fa5b1807000] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
frame=  419 fps= 31 q=-1.0 Lq=-1.0 size=    4995kB time=00:00:07.06 bitrate=5790.8kbits/s dup=210 drop=0
video:4981kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.291158%
[libx264 @ 0x7fa5b2000c00] frame I:2     Avg QP:21.72  size:142050
[libx264 @ 0x7fa5b2000c00] frame P:118   Avg QP:23.59  size: 15323
[libx264 @ 0x7fa5b2000c00] frame B:299   Avg QP:25.88  size:  1203
[libx264 @ 0x7fa5b2000c00] consecutive B-frames:  4.8%  0.0%  0.7% 94.5%
[libx264 @ 0x7fa5b2000c00] mb I  I16..4: 13.2% 60.6% 26.1%
[libx264 @ 0x7fa5b2000c00] mb P  I16..4:  0.7%  0.9%  0.2%  P16..4: 29.9%  7.9%  4.7%  0.0%  0.0%    skip:55.6%
[libx264 @ 0x7fa5b2000c00] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8: 16.0%  0.3%  0.1%  direct: 0.1%  skip:83.4%  L0:35.5% L1:63.0% BI: 1.5%
[libx264 @ 0x7fa5b2000c00] 8x8 transform intra:55.6% inter:68.7%
[libx264 @ 0x7fa5b2000c00] coded y,uvDC,uvAC intra: 50.1% 59.2% 20.9% inter: 3.3% 2.8% 0.0%
[libx264 @ 0x7fa5b2000c00] i16 v,h,dc,p: 11% 47%  9% 33%
[libx264 @ 0x7fa5b2000c00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 27% 20%  4%  6%  6%  9%  4%  9%
[libx264 @ 0x7fa5b2000c00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 25%  9%  6%  8%  7% 11%  6%  9%
[libx264 @ 0x7fa5b2000c00] i8c dc,h,v,p: 50% 29% 15%  6%
[libx264 @ 0x7fa5b2000c00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fa5b2000c00] ref P L0: 65.4% 15.1% 13.9%  5.5%
[libx264 @ 0x7fa5b2000c00] ref B L0: 88.0%  8.6%  3.4%
[libx264 @ 0x7fa5b2000c00] ref B L1: 96.4%  3.6%
[libx264 @ 0x7fa5b2000c00] kb/s:2808.83
[libx264 @ 0x7fa5b2001e00] frame I:2     Avg QP:15.35  size: 38416
[libx264 @ 0x7fa5b2001e00] frame P:108   Avg QP:20.76  size: 13229
[libx264 @ 0x7fa5b2001e00] frame B:316   Avg QP:25.94  size:  3612
[libx264 @ 0x7fa5b2001e00] consecutive B-frames:  0.9%  0.0%  1.4% 97.7%
[libx264 @ 0x7fa5b2001e00] mb I  I16..4: 33.8% 57.9%  8.2%
[libx264 @ 0x7fa5b2001e00] mb P  I16..4:  1.0%  4.2%  0.8%  P16..4: 19.9%  3.8%  2.7%  0.0%  0.0%    skip:67.6%
[libx264 @ 0x7fa5b2001e00] mb B  I16..4:  0.0%  0.2%  0.1%  B16..8: 11.4%  1.3%  0.3%  direct: 0.9%  skip:85.8%  L0:52.0% L1:42.9% BI: 5.1%
[libx264 @ 0x7fa5b2001e00] 8x8 transform intra:67.0% inter:77.7%
[libx264 @ 0x7fa5b2001e00] coded y,uvDC,uvAC intra: 38.2% 68.3% 42.4% inter: 2.3% 6.1% 1.9%
[libx264 @ 0x7fa5b2001e00] i16 v,h,dc,p: 51% 32%  8%  9%
[libx264 @ 0x7fa5b2001e00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 16% 35%  4%  5%  5%  6%  4%  6%
[libx264 @ 0x7fa5b2001e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 21% 15%  6%  7%  7%  8%  5%  5%
[libx264 @ 0x7fa5b2001e00] i8c dc,h,v,p: 42% 25% 16% 17%
[libx264 @ 0x7fa5b2001e00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fa5b2001e00] ref P L0: 71.2%  5.7% 14.7%  8.3%
[libx264 @ 0x7fa5b2001e00] ref B L0: 90.2%  7.9%  2.0%
[libx264 @ 0x7fa5b2001e00] ref B L1: 96.8%  3.2%
[libx264 @ 0x7fa5b2001e00] kb/s:2982.61
6

H.264, which is what libx264 encodes to, does not support transparency. Neither does mpeg4, the default codec for AVI. MOV output defaults to libx264 as well.

You can do this in one command:

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i images/*.png \
-filter_complex "[0:v]scale=1920x960,setdar=16:9[base];[1:v]scale=1920x960,setdar=16:9[ovr];\
 [ovr][base]blend=all_mode='overlay':all_opacity=0.7[v]"
-map [v] result.mp4

You don't need to scale both videos.If you just want the PNGs to scale the same as the base video, you can use scale2ref:

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i images/*.png \
-filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr][base]; \
 [ovr][base]blend=all_mode='overlay':all_opacity=0.7[v]"
-map [v] result.mp4

Now, you have specified a custom DAR (1920x960 is not 16:9). If that needs to be set, use

[1:v][0:v]scale2ref=iw:ih,setdar=16:9[ovr][base];

This uses a different method to apply the overlay.

ffmpeg -i original.mp4 -framerate 60 -pattern_type glob -i "images/*.png" \
-filter_complex "[1:v][0:v]scale2ref=iw:ih[ovr][base]; \
 [ovr]colorchannelmixer=aa=0.7[ovrl]; [base][ovrl]overlay[v]"
-map [v] result.mp4
  • Thank you for your detailed response. I tried your first solution as well as scale2ref and they both gave me errors: [AVFilterGraph @ 0x7fd95155c140] Error initializing threading. [AVFilterGraph @ 0x7fd95155c140] Error creating filter 'null' Error opening filters! – The Puma Mar 9 '16 at 14:59
  • Also, when trying your suggestions, ffmpeg asks if I want to overwrite the images files (each one). Why is that? They aren't changing, are they? – The Puma Mar 9 '16 at 15:00
  • See this for the error message. For the overwrite issue, can you post the command and its console output? – Gyan Mar 9 '16 at 15:48
  • I posted the command and console output. For overwrite, I just added -y to ignore the messages. – The Puma Mar 9 '16 at 16:06
  • Did you check the process thread limit for ffmpeg? Post the console output without -y. You shouldn't be getting that message. – Gyan Mar 9 '16 at 16:58

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.