Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm interested in personal analytics, and would like to find a robust way to aggregate and store my web browsing history in a format I can understand and manipulate. I mostly use Google Chrome.

For a while now I've used a Python script along these lines (which is code I got from elsewhere but I've forgotten where):

pattern = "(((http)|(https))(://)(www.)|().*?)\.[a-z]*/"
SQL_STATEMENT = 'SELECT urls.url, visit_time FROM visits, urls WHERE visits.url=urls.id;'

dumpdir = '...\Chrome History Dump'
storage = open(dumpdir+'/hdump.txt', 'a')

def date_from_webkit(webkit_timestamp):
    epoch_start = datetime.datetime(1601,1,1)
    delta = datetime.timedelta(microseconds=int(webkit_timestamp))
    return epoch_start + delta

paths = ["...\AppData\Local\Google\Chrome\User Data\Default\Archived History", 
         "...\AppData\Local\Google\Chrome\User Data\Default\History"] 

for path in paths:
    c = sqlite3.connect(path) 
    for row in c.execute(SQL_STATEMENT):
        date_time = date_from_webkit(row[1])
        url = re.search(pattern, row[0])
        try: urlc = url.group(0)
        except: urlc = "ERROR"
        storage.write(str(date_time)[0:19] + "\t" + row[0] + "\n")
    c.close()
    os.remove(path)

Unfortunately recent updates to Chrome have broken this approach---I've not looked into why yet, but I've resolved to find a better way to do this. Does anyone have any suggestions? These are some of the criteria I have in mind:

  • The solution should extract the browser history and (preferably) remove Google's copy of it.
  • It would be good if it were possible to run it automatically, either every time Chrome starts (as I did with the little script above) or on a regular schedule.
  • The solution should be as robust as possible to changes in the way Chrome stores its history--either by design, or (even better) because it is a continually maintained piece of software.
  • It need only store the URL and the visit time, the information in the history can be discarded.

Many thanks.

share|improve this question
    
It would probably be more productive to figure out how to fix your current solution for the latest version of Chrome, rather than looking for a possibly non-existing solution. –  Ivo Flipse Aug 6 '13 at 10:24
    
Well perhaps. But even when I first starting using that solution I didn't feel completely happy with it. I find it very hard to believe there isn't a better way to do this kind of thing. –  JOwen Aug 6 '13 at 20:13
1  
If you don't mind Google watching over what you're reading, you can have Google Web History track all the sites you visit and then search for metadata without burdening your own browser or computer resources mvark.blogspot.in/2013/08/… –  mvark Sep 4 '13 at 6:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.