6

Context:

Say I have two video files--rips from a DVD or DV tape. They are both MKV or AVI format, some standard container or other, but were ripped separately, with different encoders, different bitrates, and different resolutions. The videos are of different footage, but there is a segment of footage that both tracks have in common ( a segment with the same exact source material, though, as I said, the encoder/resolution/bitrate is different on each file).

For example: File 1 has a segment from 00:10 to 00:20 that has the same source material as the segment from 10:00 to 10:10 in File 2.

The Problem:

Is there a program I can feed both files into (I can remux them into different container formats first if necessary) which will tell me, even roughly, the start and end points of identical segments of the video? I.e., I could feed it the above two example files, and it would tell me the two ranges where the footage is from the same source?

Ideally, I'd want something that allows me to adjust the "confidence" of the similarity-detection, so that I could feed it, say, a really low-res, low-bitrate file and a HD file and relax its confidence so it still detected similar segments between such different-quality inputs.

The answer to this one might well be "it's impossible without a ton of work", but I figured that I'd ask anyway.

1
  • This is the kind of thing a neural network can be designed to perform, and I'm sure some software of this kind exists (the algorithm would be similar to the scene detection algorithm, but instead of comparing adjacent frames, you compare frames from different video sources). Mar 12, 2012 at 16:26

1 Answer 1

2

Interesting. I think this might be quite tractable if the size of the videos isn't too big (or you're OK with doing it in several chunks).

Here's what I'm thinking:

  • What you really want at the end is a side-by-side diff, almost like sdiff

  • But instead of line numbers, you want a time index.

  • And instead of a line of text, you want a video frame that you can compare with some level of certainty against another

I'm going to assume standard Unix tools, unless otherwise specified. They're available for every OS, including Windows.

So how about this:

  1. Extract every frame of each video to a PNG.

    You can do this with ffmpeg, see this superuser post You'll get a directory full of numbered JPGs.

  2. Make sure each one has the same aspect ratio. For this example, let's say it's 450x320. ImageMagick can help you if you don't have it.

  3. Now the tricky part. We're trying to compare frames from completely different sources, and we want to try to do without anything computationally expensive like computer vision or a neural network.

    Here's my idea: reduce each image to a tiny 1/10th size black-and-white, PNG.

    So if you had this picture of cat.jpg:

    a picture of a cat from the internet

    With a little ImageMagick: convert cat.jpg -type grayscale -resize 45x32 -depth 1 x.png it becomes: (magnified so you get the picture, no pun intended):

    enter image description here

    Now that's unique enough to be a frame signature, but not so unique (I think) that we can't get a stable checksum

  4. Repeat for every frame. Run a script like this twice, over each set of frames.

    #!/bin/sh
    for f in `ls -1` do:
        convert $f -type grayscale -resize 45x32 -depth 1 - | cksum >> 1.txt
    done
    

    So you'll get a 2.txt for the other file.

  5. Number each file. You could get fancy and make it videocode timestamps, but we'll just use nl:

    $ nl -ba 1.txt > 1n.txt
    $ nl -ba 2.txt > 2n.txt
    
  6. The grand finale. Use the highly-underrate Unix comm tool to show you what's the same between 1 and 2:

    $ comm -12 1n.txt 2n.txt
    

Try it! I bet it would work! :-)

1
  • That is bloody genius. I'll try it, and let you know how it goes. Might be a while before I get to it though. I'll come back to it. I could script those tools together, and allow the adjustment of tolerances (i.e. if I have 900/1000 frames matching across a timecode segment, that should be considered the same footage), allow the cropping of padding from aspect-ratio differences, and perhaps a "seed" value for the re-sizing settings (given two segments that I KNOW are the same, resize until they checksum to the same thing) and then I'd have something very nearly robust. Well done!
    – Zac B
    Apr 20, 2012 at 14:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .