Hi pm_vnct,
Your steps for certificate and key generation look correct. The DER format should be loaded with the `WOLFSSL_FILETYPE_ASN1` option as you are doing. The -4 option is `WOLFSSL_BAD_FILETYPE`, which would happen if the format was invalid.
Here is how the use private key buffer call should look:
wolfSSL_CTX_use_PrivateKey_buffer(ctx, buff, (long)sz, WOLFSSL_FILETYPE_ASN1);
For the connection to work you also need the peer to load the CA as trusted using the wolfSSL_CTX_load_verify_buffer API. You will also need to load the client certificate using wolfSSL_CTX_use_certificate_buffer.
Thanks,
David Garske, wolfSSL