This is a follow-up question from HERE. I want to sample k frames out of the n number of frames in a given video using FFMpeg and its thumbnail filter. There is a post here providing some insights how to select meaningful frames. However, in that question, the requirement was to select ones with scene changes higher than %40 threshold. In my case, it is opposite. I believe we should pick frames with lowest scene changes compared to their previous ones (usually in a video when something is focused, the scene is more stable and camera doesn't move much).

How to do this with FFMpeg? Can we also cluster the frames and pick representatives?

  • Do you want a command for thumbnail filter? It has just one option - cluster size – Gyan Jul 4 '18 at 5:57
  • @Gyan yes. Let's start from the command itself. I've installed the latest 2018 version N-91407. – Tina J Jul 5 '18 at 4:09

Basic command for thumbnail filter is

ffmpeg -i in.mp4 -vf thumbnail=n=100 -vsync 0 -frame_pts 1 out%d.png

This will select one representative frame out of every 100 frames.

-vsync 0 preserves the source timestamps.

-frame_pts 1 encodes that timestamp in the output filename. So, if your video is 24 fps, and output filename is out322.png, then that frame was taken from timestamp 322/24 = 13.41s of the video.

These are the relevant functions in the filter,

The filter works with packed 8-bit RGB pixel frames i.e.


Each frame's histogram is calculated as follows,

// update current frame RGB histogram
for (j = 0; j < inlink->h; j++) {
    for (i = 0; i < inlink->w; i++) {
        hist[0*256 + p[i*3    ]]++;
        hist[1*256 + p[i*3 + 1]]++;
        hist[2*256 + p[i*3 + 2]]++;
    p += frame->linesize[0];

For each pixel, three array elements' values are incremented. Their index being the color values (adjusted as per data layout shown above) of those pixel components.

Then, the average histogram of a cluster is calculated.

// average histogram of the N frames
for (j = 0; j < FF_ARRAY_ELEMS(avg_hist); j++) {
    for (i = 0; i < nb_frames; i++)
        avg_hist[j] += (double)s->frames[i].histogram[j];
    avg_hist[j] /= nb_frames;

For each component color value, their count over all frames is averaged.

Then the 'best' frame is chosen,

// find the frame closer to the average using the sum of squared errors
for (i = 0; i < nb_frames; i++) {
    sq_err = frame_sum_square_err(s->frames[i].histogram, avg_hist);
    if (i == 0 || sq_err < min_sq_err)
        best_frame_idx = i, min_sq_err = sq_err;

where the sum of squared errors is as follows

for (i = 0; i < HIST_SIZE; i++) {
    err = median[i] - (double)hist[i];
    sum_sq_err += err*err;

HIST_SIZE = 3 x 256 = 768.

  • Thanks. It works. Now what I really want is if we can select more frames from periods where camera is more stable and not moving? (so the difference between successive frames are less) – Tina J Jul 5 '18 at 13:46
  • btw, how can we change the out%d to increment one by one and then frame number instead of only showing the frame number? like out1_69.png, out2_129.png, ... – Tina J Jul 5 '18 at 20:05
  • 1
    It's either one or the other, as presently implemented. – Gyan Jul 6 '18 at 5:15
  • do u have a reference about how thumbnail really selects frames? Those histogram and whatnot – Tina J Jul 6 '18 at 13:52
  • Thanks. So basically it selects the best based on closest to the average. – Tina J Jul 6 '18 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.