I was able to recreate some of your symptoms locally. We discovered some bugs in setting up the I/O callbacks in our example server and client.

The symptom I recreated was with the error -308 happening in the wolfSSL JNI client. The I/O callback sets the timeout on socket reads, initially, to 1 second. After the handshake finishes, the client is sending "Hello wolfssl" to the server. If you have the OpenSSL server running, it should display that string. When you finally get around to typing the response to the client, the client has timed out. On a timeout, you end up with the socket error you are seeing. (Internally the handshake is complete, so the read isn't retried.)

From what I could find about the socket library, you don't actually get non-blocking sockets in Java. You can set the timeout to a very small value.

In the MyRecvCallback (or what you are using), if the Timeout exception is caught, try returning WolfSSL.WOLFSSL_CBIO_ERR_WANT_READ. Then the read should return a WANT_READ error, which will let you try the read again until something is read.