Hi ?,
That error typically means that you have not built wolfSSL with a large enough math big integer size. What size RSA key are you using?
To cover all math options add FP_MAX_BITS (max*2) and SP_INT_BITS (max). If you are using only the SP (Single Precision math) we only support 2048, 3072 and 4096, however for 4096 you need to add WOLFSSL_SP_4096.
If you can share the build settings you used (either configure or user_settings.h) that would help us provide you the best recommendation.
Thanks,
David Garske, wolfSSL