I'm trying to do some manual data extraction/encryption/decryption with the openssl command line tool.

I've been reviewing RFC5246 to work out what I need to do. It is not crystal clear to me if I am going to be able to do this step with that tool. I'm assuming the private key in the explanation is the private key generated when I created my self signed certificate.

When RSA is used for server authentication and key exchange, a 48- byte pre_master_secret is generated by the client, encrypted under the server's public key, and sent to the server. The server uses its private key to decrypt the pre_master_secret. Both parties then convert the pre_master_secret into the master_secret, as specified above.

Can someone tell me if my assumptions are correct. Can the openssl command line tool be used and supplied with my server private key and encrypted pre_master_secret from the client to generate the pre_master key for the server so it can be used to create the master key?

If so, I am not sure how to do it as I'm not very familiar with the tool.

The other thing I should point out is that the cipher suite I am working with is TLS_RSA_WITH_AES_256_CBC_SHA and I can see in Wireshark that the pre_master_secret from the client is 256 bytes long.


I'm not entirely sure but I think the answer is no. The openssl command line client is a heterogeneous collection of tools. The X.509 commands can be useful to manipulate certificates, but the cryptography commands are rarely useful for anything other than testing OpenSSL itself.

If you need to do cryptographic calculations with common algorithms, I recommend the Python interactive command line with the Cryptodome library.

But to decrypt SSL connections, the easiest way is usually to use Wireshark. Tell Wireshark where to find the private key and it will decrypt a TLS connection that uses RSA encryption. For connections using ephemeral Diffie-Hellman, you can't decrypt the traffic with the key alone, you need additional information from either the client or the server.

Note that using the TLS_RSA_WITH_AES_256_CBC_SHA ciphersuite is a bad idea for several reasons:

  • It doesn't have forward secrecy, so if the server's private key is ever compromised, all connections made with this key are also compromised. Ciphersuites that use a Diffie-Hellman key exchange (with EDH or ECDHE in their name) have forward secrecy.
  • It uses RSA decryption, which involves padding, which is a classic source of implementation bugs and leakage through side channels. Ciphersuites with EDH or ECDHE in their name in addition to RSA, or with DSA or ECDSA, use signatures instead of decryption and are less likely to suffer from implementation defects.
  • It uses CBC decryption, which involves padding, which is a classic source of implementation bugs and leakage through side channels. Ciphersuites without CBC in their name are less likely to suffer from implementation defects.

RSAES-PKCS1v1_5, which SSL/TLS-through-1.2 plain-RSA keyexchange uses, can be decrypted by OpenSSL commandline operations rsautl or pkeyutl (the latter since 1.1.0 in 2010). See their respective man pages, which should be available on your system if not Windows, or online.

Note this gives you the premaster secret, which by itself cannot decrypt (or authenticate) traffic. You must use the premaster plus nonces to derive the master secret, and then the master secret plus nonces to derive the working keys (plural). The derivation function 'PRF' differs between SSLv3 (no longer used), TLS 1.0 and 1.1 (RFCs 2246 and 4346), and TLS 1.2 (RFC 5246). (PRF will differ again in TLS 1.3 as drafted, but that will also eliminate plain-RSA keyexchange entirely.)

Commandline cannot access the SSL/TLS PRFs directly, but can do the HMACs from which they are built (except SSLv3); see the man page for dgst in the same places as above, and note -hmac $key can only handle byte sequences that can be passed from shell/etc, which is not all, so you may need -mac hmac -macopt hexkey:$hexkey instead.

That said, I concur with Gilles it's much easier to let wireshark do it. If your problem is you have the data in some form other than a capture file, wireshark distros come with several auxiliary commandline tools for manipulating files that can usually construct a fake capture, which in this case wireshark or tshark main can then decrypt.

  • Thanks I'm still on a big learning curve here. Actually trying to determine if I can do this in the MUMPS language shelling out to openssl. How do I ask a follow up question here without posting another question?
    – David B
    Jul 23 '18 at 11:31
  • (1) I don't know MUMPS (although I heard it 'shrank' to M?) so I can't say if or how it can handle the parts openssl commandline doesn't. That is probably ontopic for stackoverflow (not here). (2) The general stack philosophy is to have 1 question per Q and answers for that Q (with links if that helps explain them), so each can be found by people searching for those issues or topics. But if your followup is close to the original you might get away with editting this Q to add it (but it's impolite to remove issues already addressed). Jul 25 '18 at 1:19
  • MUMPS sometimes referred to as "M" yes you are correct. I did ask a follow up question here link] I hopefully can handle all I need to In MUMPS that openssl can not. So far so good I am trying to read the RFC and follow along though I admit I am struggling a bit. My MUMPS program is serving the certificate and I am in the process of negotiating things with the handshake message(s) it is looking good so far. If you or someone could check my follow up and assist this next that would help.
    – David B
    Jul 25 '18 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.