Not an answer to all your questions, but this is the output that my finding are based on. I might be able to share more details in a PM.

I used MSG_PEEK on the socket, to inspect the 7 received bytes, which are an Alert message rather than a ServerHello.

connect() == EINPROGRESS: Operation now in progress
epoll() EPOLLOUT: TCP socket connected.
my_IOSend() send(len=238)=238
TX: 16 03 03 00 e9 01 00 00 e5 03 03 e0 7b 69 d7 97 7c 9b c5 11 47 ce 09 b4 7a 24 68 52 8b 14 c0 9a 09 45 b3 51 8f 41 89 34 fb fa 9f 00 00 6a c0 2c c0 2b c0 30 c0 2f 00 9f 00 9e 00 ab 00 34 00 a7 00 aa cc a9 cc a8 cc aa c0 27 c0 23 c0 28 c0 24 c0 0a c0 09 c0 07 c0 08 c0 14 c0 13 c0 11 c0 12 c0 ac c0 ae c0 af 00 6b 00 67 00 39 00 33 00 16 cc 14 cc 13 cc 15 c0 06 00 b3 00 b2 c0 a6 c0 a7 cc ab cc ac cc ad c0 37 d0 01 00 b5 c0 3a 00 b4 00 45 00 88 00 be 00 c4 01 00 00 52 00 0d 00 20 00 1e 06 03 05 03 04 03 08 07 08 08 08 06 08 0b 08 05 08 0a 08 04 08 09 06 01 05 01 04 01 03 01 00 0b 00 02 01 00 00 0a 00 1c 00 1a 00 19 00 1c 00 18 00 1b 00 1e 00 17 00 16 00 1a 00 1d 00 15 00 14 01 01 01 00 00 16 00 00 00 17 00 00  (238)
nevent #1/1 has 2 events: OUT IN
available == 7
RX: 15 03 01 00 02 02 28 (7/7) PEEK
AlertProtocol (==21==0x15)
AlertLevel: 2
AlertDescription: 40 handshake_failure
my_IORecv() recv(len=5) = 5
15 03 01 00 02
my_IOSend() send(len=7)=7
TX: 15 03 03 00 02 02 46  (7)
Failed to connect wolfSSL_connect()=-1, wolfSSL_get_error(ssl, -1) = -326 (record layer version error)

Testing against the same (public) server with OpenSSL gave, which is correct:

807B8AF7FF7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40

Unfortunately, that seems like a chicken-egg scenario.

I want wolfSSL to only return the deciphered bytes from (at most) one fragment.

Interpreting 5 bytes as being a header (potentially also checking) in my receive callback is not very sound software engineering, as the state of the TLS connection is already kept inside wolfSSL. Besides, wolfSSL could potentially ask for 5 bytes even when it's not a header. Having my callback function keep state (as well as wolfSSL) sounds like overhead.

I'm mainly asking for where to modify the wolfSSL state machine to NOT try to give me all the bytes that the application requested for, but only those of one deciphered fragment. The application has no way of knowing how many bytes it should ask for, but is legal for the API contract of wolfSSL_read() to return less bytes then requested.

During the handshake, wolfSSL will not interpret an Alert message (decimal 15) correctly coming back from the server, and the error message it reports is assuming to decoding a ServerHello rather than Alert.

This happens for example when SNI is required but was not enabled by the wolfSSL client, the server responds with an Alert. wolfSSL then wrongly starts interpreting the 5 bytes as if it where a ServerHello, failing to read the actual Alert error codes itself (requiring to read a few extra bytes).

Thanks. I already implemented custom IO callbacks and these work fine, but I do not see an option or point where I could convince wolfSSL to return data for the current fragment (and leave succeeding fragments for subsequent reads to be processed).

How can  wolfSSL_read() be modified to return only the data from (at most) one SSL record?

I.e. I want to prevent wolfSSL from reading up to my 'size' argument, possibly decrypting multiple SSL records.

int ret = wolfSSL_read(ssl, (void *)buffer, size);

6

(5 replies, posted in wolfSSL)

I had the same question / request.

My use-case is mostly making the record sizes smaller if this can be negotiated.