1

Summary of the problem

I am trying to create a list (and potentially download) of all files (100s) from the following address: https://eba.europa.eu/regulation-and-policy

This is something I would use for my work on a weekly basis to identify changes or new items on the site.

The 5 clicks to do this manually for each file are as follows:

Research

I have searched for code and plugins.

What I’ve tried

I have tried using multiple plugins via Firefox and Google Chrome and also Python scripts, however, I have only been successful at downloading files and cannot generate a list of files with their attributes (where available, blank if not) e.g. topic, status, date, a link to the file etc.

I have seen this being done by a software developer so I know it can be done. This is something I would be so grateful for if you can help, thanks for taking a look!

1
  • If you can download the files, you could compare their contents for detecting changes.
    – harrymc
    Jul 29, 2020 at 19:22

1 Answer 1

1

Wget and httrack can be used to perform website strips for given files, extensions, link depths, etc. I only use wget for this purpose, though not strictly a web scraper.

wget -P [download dir] -pkr -ln [URL] - Download URL and recurse pages to link depth n and convert all links to relative filesystem.

wget -r -l4 -spider -D [URLs] [URL] - Recursively spider URL to link depth 4 restricting recursion to listed URLs.

There are some good wget guides online.

httrack is installed from Debian or Debian-based repos.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .