Here's the scenario:

  1. I have a static background png file, let's say 1920x1080
  2. I have a set of few hundreds of small png files (each with different dimensions, smth like 30x100, 10x20, 200x200 etc), each representing a changed rectangle compared to the previous frame. Each of those changes might appear in a different position, but I know those positions.
  3. there is no audio
  4. I'd like to generate a fixed, 60fps mp4 or gif from those images, using each small png as a positioned frame diff. Each frame should apply single, preassigned png on a given position for that png.

Can this be done efficiently? (I am looking mainly for time optimisation).

For example, creating a video representing a UI recording, where you have a starting frame and then each frame something small changes (cursor moves a bit, a button is pressed and changes its style), and You want to generate a full video recording, taking only small screenshot each frame, to lessen the impact of the recording.

I selected this approach as generating small png changes is possible to me and takes much less time than taking FullHD frames each time.

I am aware I could go for full, frame by frame generation, but life is too short to do it this way, especially if you want to create hundreds of such videos. I will go that way if needed, but I kinda want to push this idea as far as possible.

I'd like to do it with anything that could run on AWS Lambda, that is with limited CPU, no more than 3G of RAM, preferably with technologies like node, or python, or anything directly supported by Lambda.


I think ffmpeg is the technology you're looking for. (There are some fantastic examples and in-depth discussion here that go beyond what I'll get into.) As far as using it in AWS Lambda, it looks like ffmpeg-aws-lambda is the fastest way to get started. The example project in that repository demonstrates some basic usage. (The ffmpeg it uses was compiled with the png and mp4 codecs enabled, which is necessary for this purpose.)

Generating an mp4 from an image sequence can be done fairly easily with ffmpeg, as I'll also demonstrate shortly. The problem is that your source png files are not all the same size. You have the offset information, and there might be a way to set offsets for individual frames with ffmpeg, but if so, I haven't seen it.

ImageMagick is the go-to solution for things like "take this png, and with the offset information we have, extend its canvas with transparency so that it can be correctly composited." It is packaged for AWS Lambda here. GraphicsMagick is a much faster fork of ImageMagick which doesn't appear to be nicely packaged as a Lambda Layer, but it looks like people have made it work.

The command you'd send ImageMagick (or GraphicsMagick) to add the transparent padding to make each frame the right size would look something like this:

composite -size ${WIDTH}x${HEIGHT} -geometry +$X_OFFSET+$Y_OFFSET $IMAGE_FILE xc:transparent output_${FRAME_INDEX}.png

It might also be that you're able to generate the source PNGs with the transparent padding, which would save you that step and time and processing power.

I wrote this to generate some example pngs to play with:

#! /usr/bin/env bash
# How many frames to generate
if [ -z $LENGTH ] || [ -z ${LENGTH##[!0-9]} ]; then
  echo "Usage: $0 LENGTH"
  echo "LENGTH must be a non-negative integer"
  exit 1

mkdir -p frames

# Generate a zero-padded sequence
for i in $(seq -f "%04g" 0 $1); do
  # Some loading-icon math. The circle will grow and shrink.
  radius=$(echo "scale=1; (s($i/10) * 25) + 300" | bc -l)
  # Drawing a circle because circles are fun.
  gm convert -size 500x500 -stroke Red -fill Blue \
                           -strokewidth 2 -draw "circle 250,250 250,$radius" \
                           xc:transparent frames/frame_$i.png

Then converted to mp4 with ffmpeg:

#! /usr/bin/env bash
# -r : 60 frames a second
# -i : the frames generated above
ffmpeg -r 60 -i frames/frame_%04d.png test.mp4

ffmpeg animation from png

(To generate the above, I actually used ffmpeg -r 30 -i frames/frame_%04d.png test.gif after I discovered mp4 files cannot be attached to Stack Overflow posts.)

This still leaves the matter of compositing the transparent frames rather than merely showing them in sequence; you can find more discussion about how to do that here.

Performance-wise, I'm not sure if this will meet your needs.

[gnubeard@mothership: ~/png_to_mp4]$ time ./generate_frames 600

real    0m25.061s
user    0m23.758s
sys     0m3.036s
[gnubeard@mothership: ~/png_to_mp4]$ time ./png_to_mp4 2>/dev/null

real    0m9.763s
user    0m18.649s
sys     0m0.115s

ImageMagick is the slowest part of this process by a landslide even when I use GraphicsMagick, but that's in part because of the more time-consuming draw operation I used to make my silly circle. I changed it to a composite operation like the one I suggested above, and it was drastically faster:

[gnubeard@mothership: ~/png_to_mp4]$ time ./generate_frames 600

real    0m7.261s
user    0m5.573s
sys     0m1.738s

The good news is that the memory-intensive part, encoding the video, only takes a few hundred megabytes of RAM, so viability will depend on your time constraints and whether you can skip the ImageMagick step by generating the PNG files with the appropriate size and transparent padding.

|improve this answer|||||
  • Thank for this post, there's plenty of material to go through. I was already thinking of FFmpeg, hence the tag. Individual offsets and reducing the draw areas of each frame Is my preference, as the canvas size is impacting the operation times the most. I definitely have to fill my knowledge on compositing, as I am not sure what it does exactly. Timing looks good, but I'd like to be able to go under 3 or two seconds, especially, that Lambdas tend to take longer on its own than the average user PC. Thank you very much for the time you've put into this solution. – user12363468 Mar 26 at 7:02
  • I will have to do some more research before I can mark it as a preferable solution as it doesn't meet all of my preferences. – user12363468 Mar 26 at 7:08
  • I missed the ffmpeg tag when I first read. I'll update if I can nail down the appropriate combination of filter/overlay/compositing options necessary to achieve your desired result. For what it's worth, I ran these commands on an EC2 instance with 2 cores and 8GB of RAM. – gnubeard Mar 26 at 7:10
  • Looks like ffmpeg has some concept of lookup table (lut) filters. So far I've seen them used only for colours, but I am digging into them to see how far it can go. You can use some sort of expressions on overlay positioning, but my case would have positions which are hard to approximate with any kind of mathematical expressions. – user12363468 Mar 26 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.