Topic: [SOLVED] Getting SSL_ERROR_RX_MALFORMED_HANDSHAKE on some sites

https://www.cnet.com and https://www.pcmag.com are giving me SSL_ERROR_RX_MALFORMED_HANDSHAKE error. Can someone confirm this?

Share

Re: [SOLVED] Getting SSL_ERROR_RX_MALFORMED_HANDSHAKE on some sites

Hi renminbi,

This is a very interesting case.  These servers are sending a huge certificate message during the SSL/TLS handshake that exceeds the maximum TLS record size.  The TLS RFC states that the maximum TLS record size should be 16k.  For example, looking at the certificate message sent by www.pcmag.com:443 (IP: 192.33.31.80), it is 19097 bytes.

To work around this with wolfSSL, you will need to do two things:

1) wolfSSL limits the maximum certificate chain depth using the MAX_CERTIFICATE_SZ define.  By default this is set to about 18k.  You can increase this by defining it to a larger value at compile time using CFLAGS.  For example:

$ ./configure CFLAGS="-DMAX_CERTIFICATE_SZ=19456"

2)  Increase our internal define which limits the maximum TLS record size to 16k.  This is called MAX_RECORD_SIZE and located in <wolfssl_root>/wolfssl/internal.h.  You can increase this to something larger than the certificate message received by the server.  19k should be good (19456 bytes).

Keep in mind that increasing MAX_RECORD_SIZE creates a non-conformant build which allows records larger than the max allowed size.  A correct TLS implementation should fragment the large certificate message across multiple TLS records instead of stuffing it into a single record that exceeds the max allowed record size.

Best Regards,
Chris

Re: [SOLVED] Getting SSL_ERROR_RX_MALFORMED_HANDSHAKE on some sites

Thanks for you help.

Share