0

I have a video website. Whenever a file gets uploaded it needs to be compared to all other existing files(videos) to ensure it's not a duplicate.

Considering that with each upload the number of files to be compared gets bigger I'm looking for a solution that will be relatively fast.

Ideally this will be executed using shell_exec from php so it should return some kinda of output if the file already exists(a simple 0 or 1 will do).

I do know about cmp and diff but i fear that with the number of files existing this will be rather slow.

Are there any other solutions I should look into?

1
  • Oh, this is absolutely perfect.
    – Andrei
    Sep 23, 2019 at 10:50

2 Answers 2

2

You can compute the checksum of the file for every video and than sort the list of value. The one that have the same checksum are the same file.

In order to speedup the process, you can filter the file for the one that have a quite same size (few kb as DELTA can be enough), then you can compute the hash only for this set of file

2

Compute the checksum/hash (sha1sum) of all your videos and keep the list of checksums in a file or DB.

When you receive a new file:

  • compute its checksum/hash
  • search this checksum in the list/DB
  • if found, reject
  • if not found, accept and add the checksum to the list/DB

If you keep the checksum as an additional column in the DB, a unicity constraint on the checksum column can be used to figure out if there is already a file with the checksum and automatically fails its addition to the DB.

PS: this said, it is fairly easy to edit the comments field in a MP4 to make it look like a different video.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .