• Registered: 2020-09-27
  • Posts: 5

Topic: wolfSSL_accept fails with "record layer length error"

Hello,
I am implementing TLS 1.3 to my server application (Windows).
After the winsock accept() function, I try to attach TLS to the socket using wolfSSL_accept().
I fails with "record layer length error"and I don't understand why.

I would greatly appreciate if you can point me to the right direction.

I attach here the related code with two functions:
InitTLS_Server() is the function I use to setup TLS before listen() and accept() calls. It seems to work well.
TLS_Accept() is the function which is called after accept(). It fails.


// ________________________________________________
//
//    InitTLS_Server
//
//  PURPOSE: 
//  Initialize TLS Server settings:
//  + Select Cipher Suite
//  + Enable Client Authentication
//    + Load RSA Key
//
//  PARAMETERS:
//  None
//
//  RETURN VALUE:
//  Pointer to WOLFSSL_CTX on success
//  NULL on failure
// ________________________________________________
//
Export WOLFSSL_CTX* InitTLS_Server()
{
    WOLFSSL_CTX*    ctx;

    /* Create and initialize WOLFSSL_CTX */
    if ((ctx = wolfSSL_CTX_new(wolfTLSv1_3_server_method())) == NULL) {
        fprintf(stderr, "ERROR: failed to create WOLFSSL_CTX\n");
        return NULL;
    }

    // Select cipher to use
    if (wolfSSL_CTX_set_cipher_list(ctx, "TLS_AES_128_GCM_SHA256")  != SSL_SUCCESS)
    {
        MessageBoxA(0, "ERROR: failed to set cipher list.", "", MB_ICONERROR);
        wolfSSL_CTX_free(ctx);  /* Free the wolfSSL context object   */
        return NULL;
    }

    // Enable client authentication
    wolfSSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, 0);

    /* Load server certificates into WOLFSSL_CTX */
    if (wolfSSL_CTX_use_certificate_file(ctx, CERT_FILE, SSL_FILETYPE_PEM) != SSL_SUCCESS) 
    {
        MessageBoxA(0, "ERROR: failed to load certificate.", "", MB_ICONERROR);
        wolfSSL_CTX_free(ctx);  /* Free the wolfSSL context object   */
        return NULL;
    }

    /* Load server key into WOLFSSL_CTX */
    if (wolfSSL_CTX_use_PrivateKey_file(ctx, KEY_FILE, SSL_FILETYPE_PEM)
        != SSL_SUCCESS) {
        MessageBoxA(0, "ERROR: failed to load keyfile.", "", MB_ICONERROR);
        wolfSSL_CTX_free(ctx); 
        return NULL;
    }


    return ctx;
}


// ________________________________________________
//
//    TLS_Accept
//
//  PURPOSE: 
//  Attach TLS to socket.
//
//  PARAMETERS:
//  - socket
//  - TLS Context, which contains TLS settings such as encryption type
//
//  RETURN VALUE:
//  On success: pointer to TLS Socket (WOLFSSL*)
//  On failure: NULL
// ________________________________________________
//
Export WOLFSSL* TLS_Accept(SOCKET sckt, WOLFSSL_CTX* ctx)
{
    /* declare wolfSSL objects */
    WOLFSSL*     ssl;

    int ret;

    /* Create a WOLFSSL object */
    if ((ssl = wolfSSL_new(ctx)) == NULL) {
        fprintf(stderr, "ERROR: failed to create WOLFSSL object\n");
        return NULL;
    }

    /* Attach wolfSSL to the socket */
    if (wolfSSL_set_fd(ssl, sckt) != SSL_SUCCESS) {
        MessageBoxA(0, "ERROR: wolfSSL_set_fd error", "", MB_ICONERROR);
        return NULL;
    }

    /* Establish TLS connection */
    ret = wolfSSL_accept(ssl);
    if (ret != SSL_SUCCESS) 
    {
                // ERROR IS HERE! 
                // wolfSSL_accept fails with "record layer length error"
 
        int err = wolfSSL_get_error(ssl, ret);
        char szErr[100];
        wolfSSL_ERR_error_string(err, szErr);

        MessageBoxA(0, "ERROR: wolfSSL_accept error", "", MB_ICONERROR);
        MessageBoxA(0, szErr, "", MB_ICONERROR);
        wolfSSL_free(ssl);      /* Free the wolfSSL object */
        return NULL;
    }

    printf("Client connected successfully\n");
    
    return ssl;
}

Share

  • From: Bozeman
  • Registered: 2014-05-09
  • Posts: 777

Re: wolfSSL_accept fails with "record layer length error"

Hi Octopus01,

Thanks so much for your question. Can you compare your setup to our very simple example at the below link to see if there are any differences?

https://github.com/wolfSSL/wolfssl-exam … rver-tls.c

Also I would suggest double-checking the scope of the structures, when configuring a WOLFSSL_CTX in a function a typical flow would be like below.

int functionA(WOLFSSL_CTX* ctx) {
   // configure context
   // return success/fail code
}

int functionB(WOLFSSL* ssl) {
    // configure ssl object
    // return success/fail code
}

void functionMain(void)
{
    WOLFSSL_CTX* ctx;
    WOLFSSL* ssl;
    int return_code;

    return_code = functionA(ctx);
    return_code = functionB(ssl);
}

Warm Regards,

K

Kaleb J. Himes's Website

Share

  • Registered: 2020-09-27
  • Posts: 5

Re: wolfSSL_accept fails with "record layer length error"

Hello Kaleb,
thank you for your reply.

Upon further investigation,
I found out the exact lines of code from WolfSSL library which throw the error.

The code is contained in the wolfSSL_accept_TLSv13() function:
https://i.ibb.co/fSH6BSL/Screen-Shot-10-01-20-at-09-12-PM.png

It's failing because the client didn't send the "hello" message yet.
In fact, I didn't add TLS to the client yet.
When a client application does a connect() and then performs a send(), without initializing TLS first,
the send() call will trigger this error on wolfSSL_accept().

Problem solved. smile

Share