5

Machine A has OpenSSL 1.0.2k and I encrypt a dummy file like this:

openssl enc -aes-256-cbc -salt -in plain.txt -e -pass pass:abcd1234 -out cipher.enc

Machine B has LibreSSL 3.3.3 and when I decrypt the file from machine A like this:

openssl enc -aes-256-cbc -in cipher.enc -d -pass pass:abcd1234 -out plain.txt

It fails to decrypt, giving this error:

bad decrypt
4444794304:error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt:evp/evp_enc.c:549:

Note that if I try to decrypt it using the same command line on machine A, it works fine.

I don't know how outdated OpenSSL 1.0.2k is but unfortunately I have no control over upgrading it (it's a remote 3rd party server).

Is there an incompatibility between OpenSSL and LibreSSL when using AES encryption?

1 Answer 1

7

Found the solution, posting here in case it helps anyone else facing the same problem.

Apparently, in a later version of OpenSSL (later LibreSSL) the default hashing algorithm for key derivation was changed from md5 to sha256.

More details here: https://github.com/libressl-portable/portable/issues/378

So now I explicitly specify the hashing algo in both command lines. To encrypt on Machine A:

openssl enc -aes-256-cbc -salt -in plain.txt -e -pass pass:abcd1234 -md sha256 -out cipher.enc

And to decrypt on Machine B:

libressl enc -aes-256-cbc -in cipher.enc -d -pass pass:abcd1234 -md sha256 -out plain.txt

And now it works fine.

[Edit] -- In case the default key derivation parameters may ever change in the future, or the old implicit defaults are considered insufficient at some point, perhaps it's best to explicitly define those as well. For example by including -pbkdf2 -iter 100000.
So then it becomes:

openssl enc -aes-256-cbc -salt -in plain.txt -e -pass pass:abcd1234 -md sha256 -pbkdf2 -iter 100000 -out cipher.enc

To encrypt, and to decrypt:

libressl enc -aes-256-cbc -in cipher.enc -d -pass pass:abcd1234 -md sha256 -pbkdf2 -iter 100000 -out plain.txt

(openssl and libressl can be used interchangeably in this way)

2
  • 1
    Thanks for this! I have various Macs some with LibreSSL 2.8 and some with 3.3+ and encrypting and decrypting between the two wasn't working. I had encrypted on the old Mac (still stuck on Monterey) and couldn't decrypt on my newer Mac (which is on Ventura). Adding -md md5 to the decrypt on my newer Mac worked. Jun 27, 2023 at 15:31
  • 1
    @BenCourliss Great! I just edited my answer to add the KDF params as well, in case those ever lead to an inconsistency in future versions of either openssl or libressl.
    – RocketNuts
    Aug 2, 2023 at 12:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .