I made a few sets of certificates for s/mime encryption using openssl (for various email addresses I have). When account 1 sends an encrypted message to account 2, account 2's private key successfully decrypts the email. However, account 1's private key can ALSO decrypt the email.
I've tested this using two methods I know of that let me select a specific key to decrypt with: the android app FairEmail, and plain old openssl.
My understanding is that the sender uses the receiver's public key to encrypt the message. Therefore, the receiver's private key is the ONLY one that can decrypt the message. Based on the above, this seems not to be true. Am I missing something?
1 Answer
My understanding is that the sender uses the receiver's public key to encrypt the message
Not exactly. The sender uses the receiver's public key to encrypt a symmetric key (such as an AES or RC2 key), which is then used to encrypt the actual message.
(See "hybrid encryption" or "hybrid cryptosystem" for an explanation why.)
Therefore, the receiver's private key is the ONLY one that can decrypt the message.
The message has multiple encrypted copies of the same symmetric key – one encrypted to the recipient's public key and one encrypted to the sender's own public key.
Similarly, if you send the message to five recipients it will have a single copy of the 3DESencrypted message data, but six copies of the RSAencrypted 3DES key – one for each recipient, plus one for the sender.

I see. That makes sense. I took a look at the *.p7m and it indeed shows keys for all the recipients and the sender. Thanks!– skiwarzNov 24, 2022 at 6:48

Correct, although AES should be used instead of 3DES. CVE20162183 disclosed a major security vulnerability in DES and 3DES encryption algorithms. NIST has deprecated 3DES for new applications in 2017, and for all applications by the end of 2023. Nov 24, 2022 at 14:10

@not2savvy: While true in general, "approximately four billion blocks" makes the vulnerability extremely inapplicable to email, unless you send 256GBsized messages. As far as I've read so far, 3DES is still strong as a cipher in general (i.e. there's no specific "break" for it like there is for RC4 or singleDES), so S/MIME implementations often don't see adding AES support a priority. I think Outlook still can't read such messages? Nov 24, 2022 at 14:53

As far as I know, all MUAs use AES, but of course still support 3DES in incoming emails. I believe, that's even including Outlook! :) Nov 24, 2022 at 15:04