0

There are PDF files with thousands of lines that I need to work on a regular basis. Duplicate paragraphs are very common so I'm hoping if there's some sort of automation to find and highlight all of them.

What I usually do is copy-paste the full PDF file in Notepad ++, so what I'm looking for is a RegEx code to execute in Notepad++.

I've found a line from this post, but this is not working for my scenario.

(([^?]+\?\R(?:.+\.\R)+)[\s\S]+?)\2

I've also tried doing this in Word by using a VBA code I found, but this is taking forever to process since there's an average of 50,000 lines and 350,000 words per file. Even get a lot of not responding issue due to the amount of text it needs to process. Or maybe because my PC is old

A sample would be like this:

Although failure to release garnishment upon filing of debtor’s bankruptcy petition vio- lated automatic stay, punitive damages were not warranted because court did not find ac- tions of respondents to be egregious violation of automatic stay, nor did court find that respon- dents acted with malevolent intent.

Court need not consider whether elements of res judicata under Georgia law were met be- cause claim preclusion was inapplicable to debtor’s claim for punitive damages arising from stay violation.

Although failure to release garnishment upon filing of debtor’s bankruptcy petition vio- lated automatic stay, punitive damages were not warranted because court did not find ac- tions of respondents to be egregious violation of automatic stay, nor did court find that respon- dents acted with malevolent intent.

Court need not consider whether elements of res judicata under Georgia law were met be- cause claim preclusion was inapplicable to debtor’s claim for punitive damages arising from stay violation.

I would really appreciate all the help I can get. Thanks!

3
  • Are you concerned at all about false-positives? What I mean is in legal docs is common to use the same stock paragraphs. Do you just need to remove duplicates that are close to each other, or can you just remove all duplicate paras? What type of regex does Notepad++ use (Perl? JavaScript?)? – pbhj Nov 13 '20 at 16:31
  • Based on a search, it seems that it uses Perl. You're right, I'm only looking for paragraphs close to each other. But I'm not sure if that's the right choice since there are times duplicates happen after a span of many paragraphs. – ixcode Nov 13 '20 at 16:38
  • 1
    I'm also looking for a way to highlight only, and not replace those dupes. I just need to be aware if there are dupes and where are those, so I can change it in our CMS. – ixcode Nov 13 '20 at 16:39
1

Just replace

  • (([^?]+\?\R(?:.+\.\R)+)[\s\S]+?)\2

with:

  • (([\S\s]+\R(?:.+\.\R)+)[\s\S]+?)\2

because you haven't any question mark in your text.

  • [^?]+\? means 1 or more non ? followed by ?
  • [\S\s]+ means 1 or more any character

Rest of the explanation can be found in the linked answer.

Note: I'm the author of the linked answer.

Screenshot before:

enter image description here

Screenshot after:

enter image description here

7
  • Thanks a lot of taking the time to help me, really appreciate it. However, I tried pasting this to mark (highlight) only the duplicate paragraphs, but it's not working correctly. Can you please change the code so that it will be to highlight text instead of replacing? My goal is to be aware of duplicates so I can remove them from the CMS we use. Again thank you! – ixcode Nov 13 '20 at 16:35
  • @ixcode: A simple highlight of the duplicate is not possible, sorry. – Toto Nov 13 '20 at 16:39
  • That's ok. Just a question, I tried to do what you did there in the screenshots, but I'm getting a 0 occurrences were replaced. message. I did check Wrap around, also Regular expression. What maybe wrong in my set up? – ixcode Nov 13 '20 at 16:45
  • @ixcode: Is the sample text you gave is correct? The regex is working for the exact text you've given in the question. If the real text is quite different, please, edit your question and give a real test case. May be the regex will be different. – Toto Nov 13 '20 at 17:14
  • I forgot to attach a link of my screenshot, doing (hopefully) exactly what's in your screenshots: ma-images-dump.imfast.io/EDEN/123.png Apologies for taking too much of your time already. – ixcode Nov 13 '20 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.