1

I have a bunch of PDF files which are articles downloaded from various Journal websites. The problem is, some of these PDF files are duplicates of one another. However, if I run a basic duplicate file finder, they won't show up, because journals have the (somewhat annoying) habit of putting a cover page in the PDF before the actual article. In other words, if I have two copies of article "X" by author Y, one downloaded from site A and the other site B, they aren't exact duplicates because one has cover page from A and the other has (different) cover page from B. So my question is:

Is it possible to check if two PDF files are duplicates,ignoring the first page?

Of course, this doesn't solve all my problems, since some journal websites don't put a cover page! So the best possible question would be:

Is it possible to check if two PDF files are duplicates,where one (or both) might have a cover page?

3 Answers 3

1

According to the PDF Spec there might be some things to help you.

First of all there is the ID, a unique identifier that hould not be changed after initial creation.

From the PDF spec linked above, Section 14.4:

File identifiers shall be defined by the optional ID entry in a PDF file’s trailer dictionary (see 7.5.5, “File Trailer” ). The ID entry is optional but should be used. The value of th is entry shall be an array of two byte strings. The first byte string shall be a permanent identifier based on the contents of the file at the time it was originally created and shall not change when the file is increm entally updated

Second of all you can take a look at 7.5.6 Figure 3. According to that Figur you should have an original Body.

For the first option there is a poppler function to extract the IDs (you are interested in the permanent one) http://people.freedesktop.org/~aacid/docs/qt5/classPoppler_1_1Document.html#a2561d28a219676acff2036e5a3bacc83 , however I do not now how you could strip out the original body to compare those hashes.

1

DiffPDF can visually compare PDFs (I haven't tried it myself), also available as a portable app.

Even if you removed the cover page from all the articles with something like PDFTK, I very much doubt that the PDFs will be the same in a binary compare.

Maybe another way would be to rename each file to something unique in the articles, an article number perhaps, or a date including time? Then you could do a filename compare.

2
  • I should probably have clarified a bit: besides the cover page, the PDFs are identical. That is, before insertion of the cover page, they are the same binary file.
    – Steve D
    Apr 13, 2011 at 2:39
  • Even so, re-saving them after removal of the cover page, I doubt they will be identical. You can give it a go, but if the PDF format saves something like a date in the file after the page removal, then it will be different.
    – Hydaral
    Apr 18, 2011 at 1:54
-1

The only software I find helpful on this kind of problem is Devonthink (I am assuming you are using Mac). But you need to remove the first page from one variant. You can easily filter out the ones with the cover page by inserting a unique phrase into Finder search. Tag them, or group them, and run a script to delete the first page. When you index all of them, Devonthink will pick the duplicates. Other duplicate finders, like Gemini, Dupe Guru (is better than Gemini by the way), and others rely on some simple criteria for comparison. They don't compare the content. Devonthink does compare the content. And, it is 100% accurate, so far as I can tell.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .