1

Problem:

I can't seem to intercept HTTPS traffic for some websites using OWASP ZAP.

I have permission from the operators of the sites in question to 'scrape' those sites in order to download data from ordering & sharepoint portals.

For development/debugging purposes I have previously used OWASP ZAP as a debugging proxy, but I suspect TLS upgrades at those sites are preventing me from doing so now.


Example URL:

An example of an HTTPS URL I was previously able to load via ZAP but can no longer load is the NBN Co customer portal.

(This is is not meant for the general public, but you don't need to log in to reproduce the issue, so I feel OK giving it here.)

To be clear, my scraping code can still connect to the sites just fine, but a browser (such as firefox) via OWASP ZAP cannot do so.


Error message:

ZAP Error [javax.net.ssl.SSLException]: Received fatal alert: close_notify

Stack Trace:
javax.net.ssl.SSLException: Received fatal alert: close_notify
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2011)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1113)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375)
    at org.parosproxy.paros.network.SSLConnector.createSocket(Unknown Source)
    at org.apache.commons.httpclient.HttpConnection.open(Unknown Source)
    at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Unknown Source)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(Unknown Source)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
    at org.parosproxy.paros.network.HttpSender.executeMethod(Unknown Source)
    at org.parosproxy.paros.network.HttpSender.runMethod(Unknown Source)
    at org.parosproxy.paros.network.HttpSender.send(Unknown Source)
    at org.parosproxy.paros.network.HttpSender.sendAuthenticated(Unknown Source)
    at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)
    at org.parosproxy.paros.network.HttpSender.sendAndReceive(Unknown Source)
    at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(Unknown Source)
    at org.parosproxy.paros.core.proxy.ProxyThread.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)

The error for the example site I gave above is mentioned in a few places on the web and manifests as a 502 Bad Gateway response from the website but is returned as a 200 Ok text/plain response from ZAP to the browser. However, the message itself mentions ZAP so it's obviously not the website generating the 502.

When googling the error, most articles suggest tweaking Java or ZAP TLS protocols & ciphers and/or upgrading Java or OWAZP ZAP.

On some other sites I was previously able to tweak the proxy TLS settings to proxy some more modern TLS site configurations (i.e: in ZAP: turn off older protocols via Tools > Option > Local proxy)

I've had no luck with my example site above.


Steps to reproduce:

  1. Create and launch new empty firefox profile

firefox --no-remote -ProfileManager

Create a new blank profile called "proxy", re-select your default profile (for next startup) and then exit.

firefox --no-remote -P proxy &

  1. Install latest version of OWASP ZAP with Java 8

I'll use Docker to aid in reproduction of the issue.

Dockerfile:

FROM java:8

ENV ZAP_VERSION 2.5.0
RUN apt-get update && apt-get clean
RUN apt-get install -q -y wget && apt-get clean
RUN mkdir /zap
RUN cd /zap && wget https://github.com/zaproxy/zaproxy/releases/download/${ZAP_VERSION}/ZAP_${ZAP_VERSION}_Linux.tar.gz
RUN cd /zap && tar zxvf ZAP_${ZAP_VERSION}_Linux.tar.gz
RUN apt-get install -q -y sudo xauth libxrender1 libxtst6 libxi6 net-tools less && apt-get clean

RUN export uid=1000 gid=1000 && \
    mkdir -p /home/developer && \
    echo "developer:x:${uid}:${gid}:Developer,,,:/home/developer:/bin/bash" >> /etc/passwd && \
    echo "developer:x:${uid}:" >> /etc/group && \
    echo "developer ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/developer && \
    chmod 0440 /etc/sudoers.d/developer && \
    chown ${uid}:${gid} -R /home/developer && \
    chown -R 1000 /home/developer/ && \
    chmod -R u+rw /home/developer/

USER developer
ENV JAVA_HOME /opt/java
ENV PATH $JAVA_HOME/bin:/zap/ZAP_${ZAP_VERSION}/:$PATH
ENV HOME /home/developer

EXPOSE 8080
CMD /zap/ZAP_${ZAP_VERSION}/zap.sh -host 0.0.0.0 -port 8080 -newsession `date --iso-8601=s`

Build as:

docker build -t zaproxy:2.5.0 .

Run as:

mkdir -p $HOME/.ZAP
docker run \
  --rm \
  -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -v $HOME/.ZAP:/home/developer/.ZAP \
  -p 8080:8080 \
zaproxy:2.5.0 &
  1. Generate and save OWASP root CA

In zaproxy, Tools > Options > Dynamic SSL Certificates

Generate (if not already present), copy to text file accessible to firefox.

  1. Install saved OWASP root CA in firefox

Preferences > Advanced > Cerfificates > View Certificates > Authorities

Import > Choose your saved certificate > Open

  1. Proxy all firefox network protocols via OWASP ZAP

Preferences > Advanced > Network > Settings > Manual proxy configuration

Configure as:

  • HTTP Proxy: 127.0.0.1
  • Port: 8080
  • Tick Use this proxy server for all protocols

Then click Ok

  1. Visit https://google.com to test that the proxy is working

You should see the intercepted HTTPS requests in the lower pane of OWASP ZAP.

  1. Visit the broken URL to demonstrate the error.

1 Answer 1

0

Per zaproxy-users mailing list:

  • the remote webserver is not accepting TLS connection
  • changing "Local proxy" settings won't help but "Connection" settings might

I disabled TLS 1.1 and 1.2 in the "Connections" (outbound) options tab and I was then able to proxy that domain.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .