What is the difference between
openssl pkeyutl -sign -pkeyopt digest:sha256 and
openssl rsautl -sign ? Do they do the same thing?
pkeyutl -sign with an RSA key (and
rsa_padding_mode defaulted to
pkcs1 which specifically means pkcs1-v1_5) AND
-pkeyopt digest:$hash does steps 2-6 of EMSA-PKCS1-v1_5-ENCODE in section 9.2 of rfc3447, then step 2 of RSASSA-PKCS1-v1_5-SIGN in section 8.2.
In other words it does everything but hash the data (but it does check the length of the input hash). Note that if the hash name you specify here is different from the hash you actually used, the resulting signature will not verify correctly.
rsautl -sign (similarly defaulted) does only steps 4-6 and step 2. In other words, it does NOT add the ASN.1 AlgorithmIdentifier encoding to the raw hash, so you need to do that yourself if you want your signatures to be understood and accepted by others.
pkeyutl -sign WITHOUT
-pkeyopt digest also skips the ASN.1 encoding.
dgst -$hash -sign or its abbreviated form
$hash -sign with an RSA key does the whole job: hash, ASN.1 encode, pad 01FF..00, and modexp.