I have been trying to solve this problem for a long time, there are multiple possibilities. This gets convoluted...
Basically the activity I want to automate is, on EVERY page any browser visits - the 'right click, save as, xxx.html) action so I have a completely saved record of my browsing history.
- Shelve Add-on for Firefox does exactly this automatically in the background, and it works pretty well (it can actually use the MAFF archive html-as-a-single-file which is great, but this format slows everything down, so I just use it for HTML).
PROBLEM: I use 5 other browsers, so this isn't good enough to create a complete record.
- I have looked into Cyotek WebCopy and WinHTTrack; both are essentially web crawlers which get fed a URL and then start stripping/saving it to HTML. WinHTTTrack actually works ok, but it takes a long time (much longer than just right-click,save as, xxx.html in the browser), and worst of all it's NOT automated.
In theory what I would have to do is get my browsing history from every browser, export it as a txt file with addons (since browser histories proper are usually .sqlite files) and feed it to the program as a txt file, which it accepts, it would then go crawl that list of URLs to level 0 or 1 or whatever (you can specify the depth of the crawl in that particular program).
This is very cumbersome since it means that I can't automate the process really. I need to manually feed URLs to the program.
As an addendum to this, I investigated URL logging at the router level/setting up a web proxy so I could generate a browser agnostic URL list of all visited pages. In theory I could then feed that to the crawler, and perhaps automate it? I haven't figured out to generate this URL log yet though - there are a few programs that seem promising: Charles (a web proxy with logging), and Fiddler (a web debug/logging tool) amongst others.
A program called Proxy-offline-browser looked promising as it does EXACTLY what I want by using a web proxy to grab every URL and then saving the URL automatically, but it's apparently quite buggy, old, and slowed the browser way too much. I will try to contact the Devs to see if I'm using it incorrectly.
It also seems to only run on ONE browser at a time.
I don't think what I want should be that difficult.
In the abstract perfect what I want is this:
Every time ANY browser visits a website, the URL is logged (I think web history is sufficient, but sometimes web histories don't seem to capture every URL). Those logs from every browser are then combined into a giant list of URLs.
Then, every URL is visited by the browser automatically, saved as HTML (or a better format). And I would have it run every hour, or whenever the computer is idle, etc., on some interval.
Alternatively, I have looked into macros, since in theory if I just did the right-click save on EVERY page I visited, I would reproduce EXACTLY what I want. Macros for browsers though don't seem to work since you need to actually RUN the macro manually which defeats the purpose to me.
Does anyone know how I could do this?!