0

Background:

  • I'm trying to take an input video (which includes audio), break it into frames using cv2 (a Python library), apply some changes to the frames, and then use ffmpeg stitch them back together and add back the original audio.

My problem

  • When I go through these steps, the audio doesn't match up with the video. The video seems to be lagging behind the audio. I suspect this is happening because the original video says its framerate is 29.8218 fps, and ffmpeg may be treating that as 29 fps.

My question

  • How can I get ffmpeg to create frames at a rate of 29.8218 fps?

More information

Here's the command I use to stitch together the images:

ffmpeg -framerate X -i <path-to-input-images> -y <output>

Here's the command I use to combine the soundless video with the audio to produce the final video:

ffmpeg -i <video-path> -i <audio-path> -codec copy <output-path>

Here are the logs the two commands above produce:

ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, image2, from 'C:\Users\Nathan\Documents\rhymecraft\server\services\generate_video_for_lyrics\frames\192/%09d.png':
  Duration: 00:01:01.55, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: png, rgba(pc), 406x720, 29 fps, 29 tbr, 29 tbn, 29 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 00000227a18c10a0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000227a18c10a0] profile High 4:4:4 Predictive, level 3.0, 4:4:4 8-bit
[libx264 @ 00000227a18c10a0] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'C:\Users\Nathan\Documents\rhymecraft\server\services\generate_video_for_lyrics\soundless_videos\192\movie.mp4':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv444p, 406x720, q=-1--1, 29 fps, 14848 tbn, 29 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame= 1785 fps= 53 q=-1.0 Lsize=    4759kB time=00:01:01.44 bitrate= 634.4kbits/s speed=1.83x    
video:4737kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.453360%
[libx264 @ 00000227a18c10a0] frame I:9     Avg QP:19.92  size: 12889
[libx264 @ 00000227a18c10a0] frame P:493   Avg QP:22.73  size:  5537
[libx264 @ 00000227a18c10a0] frame B:1283  Avg QP:26.34  size:  1562
[libx264 @ 00000227a18c10a0] consecutive B-frames:  3.2%  1.6%  3.5% 91.7%
[libx264 @ 00000227a18c10a0] mb I  I16..4: 45.9%  0.0% 54.1%
[libx264 @ 00000227a18c10a0] mb P  I16..4: 14.6%  0.0%  7.8%  P16..4: 37.5% 14.2%  4.2%  0.0%  0.0%    skip:21.8%
[libx264 @ 00000227a18c10a0] mb B  I16..4:  1.4%  0.0%  0.8%  B16..8: 43.3%  5.3%  0.6%  direct: 1.0%  skip:47.6%  L0:48.9% L1:47.5% BI: 3.6%
[libx264 @ 00000227a18c10a0] coded y,u,v intra: 26.9% 10.0% 12.4% inter: 6.1% 0.9% 1.8%
[libx264 @ 00000227a18c10a0] i16 v,h,dc,p: 35% 23% 16% 26%
[libx264 @ 00000227a18c10a0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 17% 22%  5%  7%  7%  6%  5%  3%
[libx264 @ 00000227a18c10a0] Weighted P-Frames: Y:1.0% UV:0.4%
[libx264 @ 00000227a18c10a0] ref P L0: 58.3% 11.2% 20.1% 10.4%  0.1%
[libx264 @ 00000227a18c10a0] ref B L0: 90.1%  7.8%  2.1%
[libx264 @ 00000227a18c10a0] ref B L1: 95.8%  4.2%
[libx264 @ 00000227a18c10a0] kb/s:630.39
ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\Users\Nathan\Documents\rhymecraft\server\services\generate_video_for_lyrics\soundless_videos\192/movie.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:01:01.55, start: 0.000000, bitrate: 633 kb/s
    Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 406x720, 630 kb/s, 29 fps, 29 tbr, 14848 tbn, 58 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Input #1, mp3, from 'C:\Users\Nathan\Documents\rhymecraft\server\services\generate_video_for_lyrics\audio_files\64e5ed6ec1556b3cfb2fe1f39aa670cd.mp3':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    encoder         : Lavf58.27.103
  Duration: 00:01:00.03, start: 0.025057, bitrate: 192 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, s16p, 192 kb/s
    Metadata:
      encoder         : Lavc58.52
Output #0, avi, to 'C:\Users\Nathan\Documents\rhymecraft\server\services\generate_video_for_lyrics\finished_videos\192\output.avi':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    ISFT            : Lavf57.83.100
    Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), yuv444p, 406x720, q=2-31, 630 kb/s, 29 fps, 29 tbr, 58 tbn, 58 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 44100 Hz, stereo, s16p, 192 kb/s
    Metadata:
      encoder         : Lavc58.52
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 1785 fps=0.0 q=-1.0 Lsize=    6294kB time=00:01:01.46 bitrate= 838.9kbits/s speed=3.42e+003x    
video:4737kB audio:1407kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.446166%
  • Share frame extraction and stitching commands and their logs. – Gyan Sep 29 '19 at 9:23
  • @Gyan Thanks for responding; I've added the output shown when stitching the frames together, but the breaking-up of the frames doesn't produce any output, since I'm using cv2 and not ffmpeg to do it. – Nathan Wailes Sep 29 '19 at 9:26
  • What's the command? – Gyan Sep 29 '19 at 11:01
  • @Gyan I've updated the question to include the ffmpeg commands I use. – Nathan Wailes Sep 29 '19 at 11:35
  • Your video will be VFR and individual frames may have extended duration. With -framerate 29.8218, the total duration of the video stream may match but sync isn't guaranteed. COnsider using mp4fpsmod. – Gyan Sep 30 '19 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.