I'm interested in personal analytics, and would like to find a robust way to aggregate and store my web browsing history in a format I can understand and manipulate. I mostly use Google Chrome.
For a while now I've used a Python script along these lines (which is code I got from elsewhere but I've forgotten where):
pattern = "(((http)|(https))(://)(www.)|().*?)\.[a-z]*/" SQL_STATEMENT = 'SELECT urls.url, visit_time FROM visits, urls WHERE visits.url=urls.id;' dumpdir = '...\Chrome History Dump' storage = open(dumpdir+'/hdump.txt', 'a') def date_from_webkit(webkit_timestamp): epoch_start = datetime.datetime(1601,1,1) delta = datetime.timedelta(microseconds=int(webkit_timestamp)) return epoch_start + delta paths = ["...\AppData\Local\Google\Chrome\User Data\Default\Archived History", "...\AppData\Local\Google\Chrome\User Data\Default\History"] for path in paths: c = sqlite3.connect(path) for row in c.execute(SQL_STATEMENT): date_time = date_from_webkit(row) url = re.search(pattern, row) try: urlc = url.group(0) except: urlc = "ERROR" storage.write(str(date_time)[0:19] + "\t" + row + "\n") c.close() os.remove(path)
Unfortunately recent updates to Chrome have broken this approach---I've not looked into why yet, but I've resolved to find a better way to do this. Does anyone have any suggestions? These are some of the criteria I have in mind:
- The solution should extract the browser history and (preferably) remove Google's copy of it.
- It would be good if it were possible to run it automatically, either every time Chrome starts (as I did with the little script above) or on a regular schedule.
- The solution should be as robust as possible to changes in the way Chrome stores its history--either by design, or (even better) because it is a continually maintained piece of software.
- It need only store the URL and the visit time, the information in the history can be discarded.