Hi Cxdinter,
I performed tests for both wc_SignatureGenerate and wc_SignatureVerify against openssl and all passed using WC_SIGNATURE_TYPE_RSA_W_ENC with WC_HASH_TYPE_SHA256.
When using the WC_SIGNATURE_TYPE_RSA_W_ENC with wc_SignatureVerify it takes in original data being validated, hashes it, adds the DER encoding (which includes the hash OID) and compares the provided signature. This behavior is correct.
These tests were done using the code here:
https://github.com/wolfSSL/wolfssl-examples/pull/27
This was run from inside the wolfssl-examples/signature directory.
Generate a signature and compare hex output (wolfSSL vs. openssl):
./signature README.md 3 5
openssl dgst -sha256 -sign ../certs/client-key.der -keyform der -hex README.md
Result of both RSA Sign with SHA256 hex output = Matches
Sign and verify using openssl. Save sign as sign.txt.
openssl dgst -sha256 -sign ../certs/client-key.der -keyform der -out sign.txt README.md
openssl dgst -sha256 -verify ../certs/client-keyPub.der -keyform der -signature sign.txt README.md
Verified OK
Use sign.txt to verify signature using wolfSSL.
./signature README.md 3 5 sign.txt
RSA Signature Verification: Pass (0)
If you are still having the issue can you provide some examples for your openssl commands and your specific wc_SignatureVerify arguments?
Thanks,
David Garske, wolfSSL