Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a way that I can save all HTTP and HTTPS browser communications (including request, response, full headers and body) to files on my computer?

The HTTPS communications must be saved decrypted.

Ideally I'd be looking for something like an extension for Firefox or Google Chrome.

It must work on Linux.

share|improve this question

migrated from webapps.stackexchange.com Jun 17 '11 at 19:48

This question came from our site for power users of web applications.

HTTPFox is a Firefox extension that will do the job.

===EDIT===

Right click the request panel and click "Copy all rows". Then you can save it to what ever editor you like - Openoffice spreadsheet for example. Please see the image that I provided.

Answer

share|improve this answer
    
I don't think so. I've just installed HttpFox 0.8.9, and I can't see any way to save all the logged traffice to files. Please note that I have thousands of HTTP(S) requests, and I'm not willing to click a thousand times to save all of them. Please clarify how you would save all the data using HttpFox. – pts Jun 23 '11 at 16:16
    
I do think so, chief. See the edit to my answer. – slotishtype Jun 23 '11 at 16:36
1  
@slotistype: Thank you for the clarification image. I didn't know about Copy all rows before, so I tried it as you suggested. It was very easy to find it using the image you provided. Unfortunately it doesn't copy the HTTP request and response body, so it is not a solution to my question. – pts Jun 24 '11 at 23:35
    
@pts my bad. Just noticed that. I don't know of another tool, unless you intall something outside of firefox. Good luck – slotishtype Jun 25 '11 at 7:16
    
FYI HttpFox has a Content tab where I can view the response body -- except for some requests, for which it says Error loading content (NS_ERROR_DOCUMENT_NOT_CACHED). But I need the response body of all requests, so HttpFox can't be a solution to this problem. – pts Jun 25 '11 at 9:01

Fiddler is definitely something you should consider even if it is not an extension but a separate application.

share|improve this answer
    
Sorry, Fiddler doesn't work for me. because it requires Microsoft Windows. I have Ubuntu. A browser extension, as I originally asked, would probably work on multiple operating systems. – pts Jun 23 '11 at 16:07

httpFox, httpScoop (Mac only) and httpWatch (IE, Firefox, windows only) are all tools I use for doing this. Additionally you can also try Fiddler (as previosuly suggested) and also if yuo're using a Mac you could try Charles Proxy. My personal favourite is httpWatch (you can easily filter on headers, content, url etc), but this is expensive so may not be an option.

You might also want to look at the HAR project (HAR stands for HTTP Archive) and is a standards based approach to recording http transactions, headers, content etc. A number of the tools mentioned are able to export thee HAR files for viewing using he HAR Viewer (which you'll be familiar with if you've looked at the waterfall feature in Firebug). In fact, you can use Firebug to generate these files/logs for you as well using NetExport.

FWIW httpScoop is a good method for debugging/tracing HTTP traffic on a wireless network, I wrote an article about doing this some time ago: http://blog.adtools.co.uk/trace-debug-mobile-application-http-requests-using-macos/

share|improve this answer
    
Which of the tools you mention can save a dump of all logged communication to disk? httpFox can't. httpScoop is Mac-only, and I was asking for a Linux solution. httpWatch is Windows-only, and I was asking for a Linux solution. Fiddler is Windows-only, and I was asking for a Linux solution. Charles Proxy is Mac-only, and I was asking for a Linux solution. Thank you for mentioning Firebug+NetExport, that can potentially work. – pts Apr 24 '12 at 9:05
    
Firebug+NetExport work like a charm for me on Linux. They export SSL traffic unencrypted (just as I wished). I'd be happy to accept your answer, but please remove all other propositions (or move them to a separate answer) first. – pts Apr 28 '12 at 14:45

Wireshark can be used to capture network packets, including things in the http protocol layer, and save them to your computer. It can also capture https information, but I doubt that you'd be able to configure it to decrypt anything.

share|improve this answer
1  
Thank you for suggesting Wireshark, but it useless as an answer to my question, since it doesn't save the HTTPS stream decrypted. – pts Jun 23 '11 at 16:02
2  
Wireshark can decrypt SSL traffic since one of the last versions if you have the private server certificate. – cweiske Jun 29 '11 at 10:17
1  
I don't have the private server certificate. – pts Jul 2 '11 at 13:44

I found this answer wanting to do this, and none of the answers here suited me because I needed to dump HTTPS response bodies, much like OP. So, for those who land here like I did, here's what I found.

I eventually landed on mitmproxy, which isn't a Firefox extension, but does accomplish the goal of logging all Firefox traffic. It's all Python, so I installed it with pip install mitmproxy - use a package manger if you like.

Started it up with mitmproxy for initial setup/debugging, and then used mitmdump to dump the stream once I had it working and refined.

I just set Firefox to use 127.0.0.1:8080 as the proxy server (8080 being mitmproxy's default port), and I saw all my Firefox traffic coming through mitmproxy. To enable HTTPS traffic, you have to accept the certificate that mitmproxy creates - with the proxy set up, just go to the "magic address" of http://mitm.it in Firefox, click the "Other" button, check the boxes to trust the cert, and hit OK.

To dump the unencrypted response body, I had to use a very simple inline script:

from libmproxy.model import decoded

def response(context, flow):
    with decoded(flow.response):  # automatically decode gzipped responses.
        with open("body.txt","ab") as f:
            f.write(flow.response.content)

Gist here - download it as save_response.py and use it with mitmdump -s save_response.py. Response bodies will start piling up in body.txt.

mitmproxy also has a number of useful filters that you can specify to mitmdump to grab just what you need.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.