1

I have been trying to solve this problem for a long time, there are multiple possibilities. This gets convoluted...

Basically the activity I want to automate is, on EVERY page any browser visits - the 'right click, save as, xxx.html) action so I have a completely saved record of my browsing history.

  1. Shelve Add-on for Firefox does exactly this automatically in the background, and it works pretty well (it can actually use the MAFF archive html-as-a-single-file which is great, but this format slows everything down, so I just use it for HTML).

PROBLEM: I use 5 other browsers, so this isn't good enough to create a complete record.

  1. I have looked into Cyotek WebCopy and WinHTTrack; both are essentially web crawlers which get fed a URL and then start stripping/saving it to HTML. WinHTTTrack actually works ok, but it takes a long time (much longer than just right-click,save as, xxx.html in the browser), and worst of all it's NOT automated.

In theory what I would have to do is get my browsing history from every browser, export it as a txt file with addons (since browser histories proper are usually .sqlite files) and feed it to the program as a txt file, which it accepts, it would then go crawl that list of URLs to level 0 or 1 or whatever (you can specify the depth of the crawl in that particular program).

This is very cumbersome since it means that I can't automate the process really. I need to manually feed URLs to the program.

  1. As an addendum to this, I investigated URL logging at the router level/setting up a web proxy so I could generate a browser agnostic URL list of all visited pages. In theory I could then feed that to the crawler, and perhaps automate it? I haven't figured out to generate this URL log yet though - there are a few programs that seem promising: Charles (a web proxy with logging), and Fiddler (a web debug/logging tool) amongst others.

  2. A program called Proxy-offline-browser looked promising as it does EXACTLY what I want by using a web proxy to grab every URL and then saving the URL automatically, but it's apparently quite buggy, old, and slowed the browser way too much. I will try to contact the Devs to see if I'm using it incorrectly.

It also seems to only run on ONE browser at a time.

  1. I don't think what I want should be that difficult.

  2. In the abstract perfect what I want is this:

Every time ANY browser visits a website, the URL is logged (I think web history is sufficient, but sometimes web histories don't seem to capture every URL). Those logs from every browser are then combined into a giant list of URLs.

Then, every URL is visited by the browser automatically, saved as HTML (or a better format). And I would have it run every hour, or whenever the computer is idle, etc., on some interval.

  1. Alternatively, I have looked into macros, since in theory if I just did the right-click save on EVERY page I visited, I would reproduce EXACTLY what I want. Macros for browsers though don't seem to work since you need to actually RUN the macro manually which defeats the purpose to me.

  2. Does anyone know how I could do this?!

0

2 Answers 2

0

You mentioned that you use 5 web browsers. Browsing History View does a total of 4 browsers and seems to grab pretty close to every site visited but I doubt you are using Safari and IE is becoming less likely as well. It pulls from the browsers themselves so it would be a 'post browsing' save.

It may make it easier if that grabs a couple browsers and you can find something else for the others.

Edit: Technically Browsing History View actually pulls 9 histories now(8 because it has pre-IE10 and IE10/11+Edge as 2 separate options)

-1

I am in the same boat as you. The only hint of a solution I have found is a js tweak which POSTs to a local webserver when you visit a webpage. If you find anything more or have found a better solution, I would appreciate the info :) I'll keep looking also.

1
  • Please don't leave answers stating that you have the same problem as the one described in the question. If you have sufficient reputation, you may upvote the question. Alternatively, "star" it as a favorite and you will be notified of any new answers.
    – DavidPostill
    Jun 7, 2016 at 9:57

Not the answer you're looking for? Browse other questions tagged .