The rsautl OpenSSL utility does not appear to have a setting wherein the hash for OAEP can be changed. My testing suggests that it just defaults to sha1 and that's it.

My question is... is there a way to change the hash?

The command I used to test it out:

openssl rsautl -inkey publickey.txt -encrypt -oaep -in plaintext.txt -out ciphertext.txt

rsautl which is ancient, no. pkeyutl which was supposed to supersede rsautl in 2010, yes. -pkeyopt rsa_oaep_md:name or rsa_mgf1_md:name where name is a hash as in your Q on PSS.

Note for pkeyutl (edit) before 1.1.0 the order of options matters; you must do -encrypt then -inkey rsaprivate then -pkeyopt (not currently documented AFAICS, though the analogous cases in genpkey are) and you need to do rsa_padding_mode:oaep before rsa_{oaep,mgf1}_md:hash .

Same possible issue with FIPS mode.

  • 2
    I was trying this just now (maybe I tried it earlier too idk) and I had to do -pkeyopt before each option. eg. openssl pkeyutl -in ciphertext.txt -decrypt -inkey test.pem -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha1. So in that case I needed to do it once for each option - not once globally.
    – neubert
    Sep 17 '16 at 5:06
  • 1
    @neubert: yes you need -pkeyopt for each key-option; I thought that part was clear from the manpage and didn't repeat it. But the requirements on order were (and are) not all documented AFAICS, and such requirements are unusual for openssl, so I called them out. Sep 19 '16 at 6:16
  • 1
    The ordering doesn't appear to matter with OpenSSL 1.1.0f, but I couldn't find these options documented. Although -pkeyopt rsa_oaep_md:sha256 works, it is not mentioned on man pkeyutl.
    – starfry
    Jun 23 '17 at 9:04
  • @starfry: (don't know why I wasn't notified at the time) yes, 1.1.0, released 8 months after I wrote this answer, completely reimplemented option handling for all commandline operations; edited for anyone else who comes along. It's rsa_mgf1_md which applies to both OAEP and PSS -- and is documented in 1.1.1 which turnaboutly postdates your comment :-) Feb 5 '19 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.