Problem:
I can't seem to intercept HTTPS traffic for some websites using OWASP ZAP.
I have permission from the operators of the sites in question to 'scrape' those sites in order to download data from ordering & sharepoint portals.
For development/debugging purposes I have previously used OWASP ZAP as a debugging proxy, but I suspect TLS upgrades at those sites are preventing me from doing so now.
Example URL:
An example of an HTTPS URL I was previously able to load via ZAP but can no longer load is the NBN Co customer portal.
(This is is not meant for the general public, but you don't need to log in to reproduce the issue, so I feel OK giving it here.)
To be clear, my scraping code can still connect to the sites just fine, but a browser (such as firefox) via OWASP ZAP cannot do so.
Error message:
ZAP Error [javax.net.ssl.SSLException]: Received fatal alert: close_notify
Stack Trace:
javax.net.ssl.SSLException: Received fatal alert: close_notify
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2011)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1113)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375)
at org.parosproxy.paros.network.SSLConnector.createSocket(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.open(Unknown Source)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Unknown Source)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(Unknown Source)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.parosproxy.paros.network.HttpSender.executeMethod(Unknown Source)
at org.parosproxy.paros.network.HttpSender.runMethod(Unknown Source)
at org.parosproxy.paros.network.HttpSender.send(Unknown Source)
at org.parosproxy.paros.network.HttpSender.sendAuthenticated(Unknown Source)
at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)
at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)
at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)
at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
The error for the example site I gave above is mentioned in a few places on the web and manifests as a 502 Bad Gateway
response from the website but is returned as a 200 Ok
text/plain response from ZAP to the browser. However, the message itself mentions ZAP so it's obviously not the website generating the 502.
When googling the error, most articles suggest tweaking Java or ZAP TLS protocols & ciphers and/or upgrading Java or OWAZP ZAP.
On some other sites I was previously able to tweak the proxy TLS settings to proxy some more modern TLS site configurations (i.e: in ZAP: turn off older protocols via Tools
> Option
> Local proxy
)
I've had no luck with my example site above.
Steps to reproduce:
- Create and launch new empty firefox profile
firefox --no-remote -ProfileManager
Create a new blank profile called "proxy", re-select your default profile (for next startup) and then exit.
firefox --no-remote -P proxy &
- Install latest version of OWASP ZAP with Java 8
I'll use Docker to aid in reproduction of the issue.
Dockerfile:
FROM java:8
ENV ZAP_VERSION 2.5.0
RUN apt-get update && apt-get clean
RUN apt-get install -q -y wget && apt-get clean
RUN mkdir /zap
RUN cd /zap && wget https://github.com/zaproxy/zaproxy/releases/download/${ZAP_VERSION}/ZAP_${ZAP_VERSION}_Linux.tar.gz
RUN cd /zap && tar zxvf ZAP_${ZAP_VERSION}_Linux.tar.gz
RUN apt-get install -q -y sudo xauth libxrender1 libxtst6 libxi6 net-tools less && apt-get clean
RUN export uid=1000 gid=1000 && \
mkdir -p /home/developer && \
echo "developer:x:${uid}:${gid}:Developer,,,:/home/developer:/bin/bash" >> /etc/passwd && \
echo "developer:x:${uid}:" >> /etc/group && \
echo "developer ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/developer && \
chmod 0440 /etc/sudoers.d/developer && \
chown ${uid}:${gid} -R /home/developer && \
chown -R 1000 /home/developer/ && \
chmod -R u+rw /home/developer/
USER developer
ENV JAVA_HOME /opt/java
ENV PATH $JAVA_HOME/bin:/zap/ZAP_${ZAP_VERSION}/:$PATH
ENV HOME /home/developer
EXPOSE 8080
CMD /zap/ZAP_${ZAP_VERSION}/zap.sh -host 0.0.0.0 -port 8080 -newsession `date --iso-8601=s`
Build as:
docker build -t zaproxy:2.5.0 .
Run as:
mkdir -p $HOME/.ZAP
docker run \
--rm \
-e DISPLAY=$DISPLAY \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v $HOME/.ZAP:/home/developer/.ZAP \
-p 8080:8080 \
zaproxy:2.5.0 &
- Generate and save OWASP root CA
In zaproxy, Tools
> Options
> Dynamic SSL Certificates
Generate
(if not already present), copy to text file accessible to firefox.
- Install saved OWASP root CA in firefox
Preferences
> Advanced
> Cerfificates
> View Certificates
> Authorities
Import
> Choose your saved certificate > Open
- Proxy all firefox network protocols via OWASP ZAP
Preferences
> Advanced
> Network
> Settings
> Manual proxy configuration
Configure as:
HTTP Proxy
: 127.0.0.1Port
: 8080- Tick
Use this proxy server for all protocols
Then click Ok
- Visit
https://google.com
to test that the proxy is working
You should see the intercepted HTTPS requests in the lower pane of OWASP ZAP.
- Visit the broken URL to demonstrate the error.