I have installed openssl and now the rsa_test.c is running fine. What I want to do is:
Open any exe or dll digital certificate. Extract the Thumbprint and PublicKey. The public key contains the exponent too (if you are familiar with ASN1 encoding).
Now I have already calculated SHA-1 hash of the file and compared it to the MD field to find it correct. So now I want to pass the Thumbprint, PublicKey->m and PublicKey->e to RSA_public_decrypt function to calculate the decrypted SHA-1 hash of the file.
What Padding Algorithm should I pass to the function? When I pass RSA_NO_PADDING, it decrypts but give me the 128-bytes long hash which is not at all correct. For all other padding algorithm options it returns the error of padding not found.
Would you please teach me or tell me the correct parameters to pass through the function with an example or good link to example?
