2

What would be the best tool / method to automatically find a position from an video file, where the frame approximately matches* a given image?

Basically, a command to: "Find the position where [image.jpg] appears in the [video.mpg]"

Preferably with ffmpeg, or some other linux command-line tool.

* - or maybe the frame which best of all of the frames in the video, matches the image

  • If something like this exists, then it's really cool – That Brazilian Guy Oct 22 '13 at 20:58
  • I would think there are some image matching libraries somewhere - combining those with a tool that extracts frames from video shouldn't really be that hard. If such doesn't exist, I'm quite tempted to write one myself. :) – Ilari Kajaste Oct 22 '13 at 21:06
  • There are some amazing image matching tools out there! (Although this one is not for linux, it will compare different image formats, dimensions, color depths, and even partial images; so maybe there is a similar CLI tool or library for linux that you can pipe outputs?) – That Brazilian Guy Oct 23 '13 at 0:56
  • So, have you wrote such a tool yourself? Would've been really useful. – user Jun 2 '14 at 14:32
  • @user3075942 Not yet, but the concept is still on my todo. :) – Ilari Kajaste Jun 2 '14 at 14:33
1

framepos.sh is a bash script to search images in a video file, using ffmpeg and findimagedupes.

with the threshold variable, you can specify the fuzziness of the search.

we use the script with success to remove intros and outros from tv show episodes.

for "audiodetect" we use dejavu, and maybe pyAudioAnalysis Segmentation can be used.

relevant pseudocode ....

most complexity in the code is to run "extract" and "compare" in parallel, to improve speed and efficiency:

function framepos() {

    # extract frames. run in background
    # bmp format is faster than png or jpg
    $ffmpeg_cmd \
        $ff_args -i "$V" \
        ${tmp_pre}frame-%04d.bmp \
        2>/dev/null &
    pid=$!

    # output function for findimagedupes
    script_include=$(cat <<-'EOF'
        VIEW () {
            for f in "$@"
            do
                echo -n "$f"
                echo -ne "\t\t\t\t"
            done
            echo
        }
EOF
    )

    n2=0
    while true
    do
        n=$(ls ${tmp_pre}frame-*.bmp 2>/dev/null | wc -l)

        (( $n == 0 )) && {
            kill -0 $pid 2>/dev/null || {
                # extract done
                echo debug found no match >&2
                break
            }

            kill -SIGCONT $pid
            sleep $step_duration
            continue
        }
        (( $n == 1 )) && {
            # only one frame extracted.
            # if ffmpeg is still extracting, this file is incomplete
            kill -0 $pid 2>/dev/null && {
                # extract running
                kill -SIGCONT $pid
                sleep $step_duration
                continue
            }
            n2=1
        }
        (( 1 < $n && $n <= $frames_bufsize )) && {
            # frame buffer not full
            # if extract is running, then wait before compare
            kill -0 $pid 2>/dev/null && {
                # extract running
                kill -SIGCONT $pid
                sleep $step_duration
                continue
            }
            n2=$(( $n - 1 ))
        }
        (( $n > $frames_bufsize )) && {         #echo found $n frames
            # pause frame extraction to save space
            # extract is faster than compare
            kill -SIGSTOP $pid
            n2=$(( $n - 1 ))
        }

        echo compare $n2 frames
        break_while=false
        for I_cur in "${I[@]}"
        do
            # we need the "real path" for findimagedupes
            pattern=$(readlink -f "$I_cur")

            # call findimagedupes
            # to find "visually similar images"
            res=$(
                ls ${tmp_pre}frame-*.bmp \
                | head -n $n2 \
                | xargs findimagedupes -t $threshold \
                    -i "$script_include" "$pattern" \
                | grep -a "$pattern"
            )

            if [ ! -z "$res" ]
            then
                res=$(
                    echo "$res" \
                    | sed 's/\t\t\t\t/\n/g' \
                    | grep -v '^$' \
                    | grep -v "$pattern" \
                    | sort \
                    | head -n 1 \
                    | sed -E 's/^.*frame-(.*?)\.bmp$/\1/'
                )

                # get frame time
                # note: frame numbers start with 1
                # frame minus one:
                t=$(
                    echo $T1 $res $fps \
                    | awk '{printf "%.4f\n", $1 + ( ( $2 - 2 ) / $3 ) }'
                )
                # matching frame:
                #   | awk '{printf "%.4f\n", $1 + ( ( $2 - 1 ) / $3 ) }'

                # return
                echo $t

                # stop extracting
                kill -9 $pid 2>/dev/null

                # remove all temp files
                rm ${tmp_pre}frame-*.bmp

                break_while=true
                break
            fi
        done

        $break_while && break

        # remove processed temp files
        (( $n2 > 0 )) \
        && ls ${tmp_pre}frame-*.bmp | head -n $n2 | xargs rm
    done
}
0

You can use the md5 and framemd5 muxers if the image and frame are exactly the same. Example:

  1. Get MD5 sum of target image:

    $ ffmpeg -i frame.jpg -f md5 - 2>&1 | grep MD5
    MD5=b7fb5124a65108ebb067129d9d81ed57
    
  2. Find exact image frame in video:

    $ ffmpeg -i video.mov -f framemd5 - 2>&1 | grep b7fb5124a65108ebb067129d9d81ed57
      0,        62,        62,         1,  1424400, b7fb5124a65108ebb067129d9d81ed57
    

The output numbers refer to: stream_index, packet_dts, packet_pts, packet_duration, packet_size, MD5.

However, this is probably not what you're looking for. Conceivably this will work if video.mov is mjpeg; such as if you make a video from jpg inputs and stream copy them to the output:

ffmpeg -pattern_type glob -i "*.jpg" -codec copy output.mkv
  • You're right, exact match is not what I'm looking for. – Ilari Kajaste Oct 23 '13 at 4:26
  • @IlariKajaste Consider submitting a feature request to the FFmpeg Bug Tracker. Mention any libraries that may do what you want if you find any. – llogan Oct 23 '13 at 5:51
  • A feature like this might be a bit out of scope as direct feature of FFmpeg. But with a small script and some piping this might be possible. I don't know, though, which is why I'm asking. :) – Ilari Kajaste Oct 23 '13 at 6:58
  • 1
    @IlariKajaste There are already several "detect" filters in ffmpeg: silencedetect, volumedetect, blackdetect, blackframe, cropdetect, edgedetect, so a "framedetect" may not be out of scope. – llogan Oct 23 '13 at 17:54
  • Ah, yes, good point! – Ilari Kajaste Oct 24 '13 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.