Hi ajay,
VERY INTERESTING!
Ok first the fix, then the details. Since you are building on a 32-bit system ECC is disabled by default so only DHE_RSA cipher suites are on by default.
The short-term solution is to do this:
git clone <wolfssl>
cd wolfssl
./autogen.sh
./configure --enable-ecc
make
./examples/client/client -h device.instamsg.io -p 8883 -x -d -g
Details:
Ok so after doing some more tests against that endpoint here's what I've found.
If you use a DHE_RSA cipher suite rather than sending back a "Server Hello Done" message at the end of the handshake, that endpoint responds with this plaintext message.
use only10UVeriSign Trust NetworkN0L10 UFR10U
KEYNECTIS1
0UROOT10UKEYNECTIS ROOT CA
So wolfSSL, which is looking for a TLS packet that is encoded as the SERVER HELLO DONE message, thinks it is a malformed message. Even though it is plaintext the TLS record indicates it is an "Encrypted Hanshake Message" which is in fact technically a malformed buffer at that point in the handshake so the error is correct.
The reason openSSL s_client is able to recover from this is some form of renegotiation. We see the following:
WOLFSSL - reports error at this point and aborts the connection
OPENSSL - Sees strange packet and re-sends Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message all over again.
device.instamsg.io replies to this with - Change Cipher Spec, Hello Request, Hello Request (These are "finished" messages).
If instead we connect to that end-point with an ECDHE_RSA cipher suite (instead of DHE_RSA) then everything is normal. We are unsure why that end-point is failing to send a SERVER HELLO DONE, our best guess would be an implementation bug of some sort except that it is a plain text message so it seems to have a purpose.
It is almost as if this is a custom protocol similar to start-TLS or something along those lines where the plain text message has some design behind it. OpenSSL either supports this custom protocol or is able to get around it by doing some sort of re-negotiation.
You stated this is a proprietary server of yours so perhaps you can shed some light on the matter. Can you explain what this replacement message is for the SERVER HELLO DONE message and what it's purpose might be?
For your use I am attaching two wireshark traces showing our findings.
This is the openSSL command we used for testing:
openssl s_client -connect device.instamsg.io:8883 -cipher DHE-RSA-AES128-GCM-SHA256
Everything will work fine with an ECDHE cipher suite, it's only the DHE cipher suites that are behaving abnormally in a manner we do not yet support.
Warmest Regards,
Kaleb