Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

There is a great program for windows (URL Snooper: http://www.donationcoder.com/Software/Mouser/urlsnooper/index.html) that allows you to view all URLs being requested on the machine.

Does any such program exist in linux (preferably command line)

share|improve this question
    
im not sure what is that you are trying to do here. –  gweg Nov 8 '09 at 8:41
    
Actually, it seems that URL Snooper does not only apply to URLs being requested on the machine, but also to URLs hidden in the HTML source of some page, which are not necessarily requested yet. For the latter see also "How to download list of files from a file server?" at superuser.com/questions/47089/… –  Arjan Nov 8 '09 at 13:00
add comment

migrated from stackoverflow.com Nov 8 '09 at 8:55

This question came from our site for professional and enthusiast programmers.

4 Answers

up vote 8 down vote accepted

It seems that URL Snooper does not only apply to URLs being requested on the machine, but also to URLs hidden in the HTML source of some page, which are not necessarily requested yet. For the latter see also "How to download list of files from a file server?" here at Super User. Or, in Firefox see menu Tools » Page Info » Media, or use add-ons like Video DownloadHelper or UnPlug. The following applies to seeing all URLs that are actually requested.

The command line ngrep could do it, but gives far more details then you'd probably want.

Like: it will not simply show you the URL as typed into the location bar of a browser, but the whole HTTP request. (So: the IP address as resolved by the browser before actually making the request, and then the HTTP request the browser sends to that IP address.) And: it will also show this for every image etcetera used in the resulting page.

You might need to install ngrep, like on a default installation of Ubuntu:

sudo apt-get install ngrep

To capture all HTTP GET requests to port 80:

sudo ngrep -W byline -qilw 'get' tcp dst port 80

Still, that would show you the whole request. (Try for yourself, if you're a Super User!) To limit that output some more to show only lines with -> , get or host:

sudo ngrep -W byline -qilw 'get' tcp dst port 80 \
  | grep -i " -> \|get\|host"

Or, to capture all requests to port 80, but ignore those with the Referer header set (as set when requesting embedded images etcetera, but also set when clicking a link in a web page, thus only showing requests that are typed into a browser's location bar directly, or are opened in a new window, or are opened from a bookmark or email):

sudo ngrep -W byline -qilwv 'referer' tcp dst port 80 \
  | grep -i " -> \|get\|host"

Also sniffer tools like Wireshark have command line options. And, just as an aside and far more more basic, tcpdump is installed on most Linux distributions:

sudo tcpdump -Alfq -s 1024 \
  'tcp dst port 80 and ip[2:2] > 40 and tcp[tcpflags] & tcp-push != 0' \
  | grep -i " > \|get\|host"
share|improve this answer
add comment

I can also recommend url-sniff by Pawel Pawilcz. It is a lightweight Perl script that wraps nicely around ngrep. It supports colorized output as well. Here you will find a screenshot. It gives you a simple interface for sniffing all requested URLs.

share|improve this answer
    
The url-sniff website is offline, but there is a copy in the archive. –  Stefan Schmidt Feb 6 '12 at 13:36
add comment

You could use an HTTP proxy such as Privoxy, but you'll have to configure your browser to use it - it doesn't snoop network traffic. It makes a log of URLs accessed which you can view with a text editor.

share|improve this answer
add comment
  1. Run a proxy that logs the requests as suggested by navilon. You don't need to configure anything on client side if you go for a transparent proxy.
  2. Run a sniffer on the gateway.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.