I am trying to remove Wayback toolbar code using some tools such as find & xargs and grep with sed etc

issue that the size of the html & php files it's over 7GB & as well as the pages over (3 millions page) I want to do it for all pages that begin with Wayback tooldbar which I've downloaded from web.archive.org. the good thing that the wayback toolbar code is start & end with

 Wayback tooldbar code
 bla bla bla 

but no order in which line..So, i try with the below command

find . -type f -name '*.php*' | xargs -I {} grep -l '<!-- BEGIN WAYBACK TOOLBAR INSERT -->' '{}'

i can catch those files that contain the toolbar

but I've stuck in the how to remove the code in between < start and end>


You'll want to use sed to delete that stuff:

find ... -print0 | 
| improve this answer | |

You can retrieve an unmodified copy of HTML documents from the Wayback Machine by appending id_ to the snapshot date/time in the URL.

For example:

With toolbar: https://web.archive.org/web/20180101010338/http://www.example.com:80/

Raw original: https://web.archive.org/web/20180101010338id_/http://www.example.com:80/

| improve this answer | |

On macOS:

find ./ -type f -exec sed -i '' -e '/BEGIN WAYBACK TOOLBAR INSERT/,/END WAYBACK TOOLBAR INSERT/d' {} \;

macOS specifically needs an empty string passed in as an argument to -i because they use the BSD version of find instead of the GNU version on linux so there are some differences. I can't find the stackoverflow post on it now, but I'm sure it could be searched for.

| improve this answer | |
  • This looks fairly similar to the accepted answer. Please explain what you changed and why. – G-Man Says 'Reinstate Monica' May 12 '19 at 18:46
  • macOS find tool requires an extra argument, the double quotations ('') in order to work. Using the first answer, you encounter the following error: find: ...: No such file or directory. The find0 argument looks incorrect as well (or at least I was not able to get it working). There was another answer (that I cannot find now) that says you have to pass an empty string to the -i function specifically on macOS. Using this answer, you are able to traverse through all the files. It is unfortunate that the working answer is downvoted. – Snuupy May 13 '19 at 20:09
  • (1) In English (and a bunch of other languages, I believe), “...” means “and so on”, “whatever”, “blah blah blah”, or similar things — it’s a placeholder.  It’s a shame that glenn used ... as a placeholder for . -type f -name '*.php*' (or whatever find arguments you want to use), but I’m sure he didn’t mean for users to type “...” literally.  (2) If you typed “find0”, it’s no wonder it didn’t work, because glenn didn’t say “find0”.  (3) If you would edit your answer to include the explanation you just posted in the above comment  … (Cont’d) – G-Man Says 'Reinstate Monica' May 13 '19 at 20:55
  • (Cont’d) …  (specifically, the part(s) about sed -i, maybe stated in a bit more detail, and not the parts about ... and “find0”, which are just reflections of your misunderstanding, and maybe add something about -exec vs. xargs), it might get upvoted.  (4) find . is just as good as find ./.  I would advise against using find ./ because, if you get into the habit of typing that, it’s too easy to type find / by accident.  Murphy’s Law says that you will make that error at a time when it will cause the most damage possible. – G-Man Says 'Reinstate Monica' May 13 '19 at 20:55
  • updated original answer to include an explanation. – Snuupy May 14 '19 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.