30

I'm using the Charles Web Debugging Proxy software to debug HTTP requests. It works great with my desktop browsers, Chrome and Firefox, and it even sees HTTP requests that other programs make.

When I run Charles and check the network config, I think I understand how it works - it just sets up a proxy for all HTTP and HTTPS requests and then listens for these on port 8888:

enter image description here

However the item I can't figure out is that I don't see any requests that I initiate at the Terminal, such as wget, curl, or the elinks browser.

I know that I can specify the proxy with curl and wget by using 127.0.0.1:8888, but I don't understand if the network interface is set up with a proxy in the configuration why I would need to manually specify the proxy for them.

Also I can't seem to get BlueCrab (website copier) to show up in Charles either - and I don't see a proxy setting for it - although I believe it is using an XWindow wrapper or something (so it's not really a native Cocoa / Carbon app):

enter image description here

How can I get all HTTP requests on my system to run through Charles?

Clarification

My question is about the system fundamentals of why curl and wget wouldn't use a proxy when the network interface was set up to use one more so than asking about the correct syntax for curl, wget, etc.

Improve this question

4 Answers 4

Reset to default
4

VPN would create a new network device, you can see it in ifconfig command, and then route all system network to that device, you could see the route use route command.

But HTTP Proxy (in this case, the Charles) is different, it just open a port, to use it, you must specify your application setting to use that port for HTTP stuffs. and as Kevin Reid's answer, curl, wget etc don't read OS X's system wide settings.

If your proxy is SOCKS (Charles supports both HTTP and SOCKS), you could use ProxyChains or tsocks for application doesn't support proxy setting.

e.g.:

$ proxychains git clone https://github.com/rofl0r/proxychains-ng
Improve this answer
3
  • How to have a "VPN-like" Web debugging then? Are there apps that do this?
    – Pacerier
    May 21, 2015 at 6:28
  • @Pacerier Try Wireshark, it's an Internet / network monitoring, analyse tool, in which you could view pretty much all the network traffics of your system, include but not limited to UDP, TCP (includes HTTP). However, it's way too complicated for web debugging, but enter a one-liner filter command so only HTTP traffic displays is possible.
    – Mengdi Gao
    May 21, 2015 at 15:45
  • @Pacerier also consider mitmproxy, when running in Transparent Proxy mode, you got VPN-like web debugging. The down side is that it's a command line tool and configuration is a bit complicated.
    – Mengdi Gao
    May 21, 2015 at 15:58
21

wget behind proxy (you may have to create the rc file) source

`$ vim ~/.wgetrc`

Add the following line:

http_proxy=http://127.0.0.1:8888

curl behind proxy source

$ vim ~/.curlrc

Add the following line:

proxy = 127.0.0.1:8888

elinks behind proxy source

Find your elinks.conf file with:

sudo find / -name elinks.conf

Add the following line:

protocol.http.proxy.host "127.0.0.1:8888"

Not sure about BlueCrab

Improve this answer
4
  • 1
    > I know that I can specify the proxy with curl and wget by using 127.0.0.1:8888, but I don't understand if the network interface is set up with a proxy in the configuration why I would need to manually specify the proxy for these things.
    – cwd
    Mar 10, 2012 at 0:08
  • Your question title and what you were looking for don't match. I will try to find that info as well, though.
    – Rebecca Dessonville
    Mar 10, 2012 at 0:15
  • 1
    I think it does match - I am looking for a way to run ALL requests through Charles Debugging Proxy and not have to set up rules on a program by program basis. Don't worry if this question goes stale and nobody else can answer it I will come back and accept your answer. And thank you for sharing the config syntax for those utilities.
    – cwd
    Mar 10, 2012 at 0:27
  • 1
    After reading your edit I now see where I misunderstood your original request.
    – Rebecca Dessonville
    Mar 10, 2012 at 3:50
12

The reason why you don't just get proxying of all HTTP requests is because at the operating system level, there is no such thing as a "HTTP request"; there are only TCP connections. Contacting a HTTP proxy means changing the HTTP request slightly as well as contacting the proxy server instead of the host named in the URL, so it has to be done in the code that implements sending HTTP requests.

curl and wget have their own HTTP code, which uses their own config files — they haven't been programmed to look for proxy settings where Mac OS X keeps them, nor are they using the HTTP libraries provided with Mac OS X that use those proxy settings.

Improve this answer
9

If you dont want to touch your configuration files, using curl you can do :

curl http://example.com --proxy 127.0.0.1:8888

Improve this answer
1
  • This is the best answer if it is curl that you specifically want to proxy
    – Mark Tickner
    Apr 12, 2022 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.