SSL: curl works, httpie fails

Question

I am running on Ubuntu 16.04.4 LTS. I have the following versions:

root@e816b85d954d:/# http --debug
HTTPie 0.9.9
Requests 2.9.1
Pygments 2.1
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609]
/usr/bin/python
Linux 4.4.0-116-generic

root@e816b85d954d:/# curl --version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets 

curl is working fine:

$ curl https://mysite

But httpie is failing:

root@e816b85d954d:/# http https://mysite

http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) while doing GET request to URL: https://mysite

Why could this be the case?

Answer 1

The error message is pretty straight-forward. It's telling you that the web server contains a SSL certificate which has some problem (most probably because it's self-signed, it's expired etc.).

You should be able to stop checking the certificate validity with the --verify option:

--verify VERIFY

Set to "yes" to check the host's SSL certificate. You can also pass the path to a CA_BUNDLE file for private certs. You can also set the REQUESTS_CA_BUNDLE environment variable.

So, setting --verify no shall be enough.

# http --verify no https://...
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
[...]

It's always best recommended to use a valid SSL certificate, though.

Answer 2

Better late then never :) For the future generations:

I just encounter (and solved) similar problem. In my case I'm not using self-signed certificate but I thing neither were you.

In cryptography and computer security, a self-signed certificate is a security certificate that is not signed by a certificate authority (CA)

Source: https://en.wikipedia.org/wiki/Self-signed_certificate

I think you meant signed by private-CA, which will not be respected out of box by any public OS.

In my case the problem was that the server returned only Intermediate certificate instead of full chain (RootCA was not included), with other servers that are using the same certification but included rootCA both curl httpie worked fine.

My problem was like this:

Let my.private.api.net be a host that returns only Domain certificate my_api.pem that is signed by Intermediate certificate SubCA.pem which in turn is signed by RootCA.pem.

I'm using "raw" ubuntu, i mean nothing was done with CA_BUNDLE etc.

I have both SubCA.pem and RootCA.pem files, verifing against RootCA won't work as expected, this is what happens to me when using SubCA.pem

~> curl https://my.private.api.net --cacert SubCA.pem
OK.

but

 ~> http https://my.private.api.net --verify SubCA.pem

http: error: SSLError: HTTPSConnectionPool(host='my.private.api.net', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate
verify failed')])"))) while doing a GET request to URL: https://my.private.api.net/

So this does looks like op's case.

As I mentioned before if the admin would change the server setup (to return all certs during handshake) it would fix the problem.

However I found a way to fix this without him. Validating against any of the certs won't work but validating against both will!

What we need is so called "Bundled" certificate, which simply one file with both certs stack on each other, so we create a new file, lets name it Bundled.pem that looks like this:

-----BEGIN CERTIFICATE-----
<<content from SubCA.pem>>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<<content from RootCA.pem>>
-----END CERTIFICATE-----

order is important!

now httpie works fine.

~> http https://my.private.api.net --verify Bundled.pem
HTTP/1.1 200 See Other
...

OK.

and so does curl:

~> curl https://my.private.api.net --cacert Bundled.pem
OK.