After much head-smashing I was able to get the compiler to work. It turns out there are two different #defines of:
SIZEOF_LONG
SIZE_OF_LONG
SIZEOF_LONG_LONG
SIZE_OF_LONG_LONG
Enabling both "#define CYASSL_SHA384" and "#define CYASSL_SHA512" the connection with httpbin.org using SHA384 certificates fails on verifying the signature:

About to verify certificate signature
SigOID 656
Rsa SSL verify match encode error
Confirm signature failed
Failed to verify CA from chain

It seems as if the encoding verify fails when checking the certificates from the server. Returning error 155: "ASN sig error, confirm failure"
Trying to remove the intermediate certificates to not be individually verified (loading only the root certificate in the chain), it then fails on error 188 (No CA signer to verify with ASN_NO_SIGNER_E) for not having the intermediate certificates (though the server sends all 4 certificates in the verification chain)

Also, have you run the ctaocrypt test on your target board? It does some known answer tests for all the ciphers and hashes you have enabled.

Looking more at the test SHA384 code, it looks like it is failing to get the correct memcmp() value returned.

if (memcmp(hash, test_sha[i].output, SHA384_DIGEST_SIZE) != 0)
            return -10 - i;

Checking the values of things:

SHA384_DIGEST_SIZE = 48

After InitSha384(&sha):
hash: 20017E30
test_sha[0].output 0

After Sha384Update(&sha,...):
hash: 20017E30
test_sha[0].output 801A4D4

After Sha384Final(&sha, hash):
hash: 20017E30
test_sha[0].output 801A4D4

For the "client" command you can only include one certificate with the "-A" option. The command line option processor saves the file name and loads the file later, so only the last file name is saved. If you

cat AddTrustCert ComodorRSACertAuth ComodorRSADomainValidSecureServerCA > combo.pem

and then -A combo.pem, all three will be loaded. (This assumes the certificates are in PEM format. We can't load DER certificates concatenated together.)

Back to the SHA-384 issue. Do you have 64-bit data types available on your platform? Could you do a

printf("sizeof(long long) = %u\n", sizeof(long long));

The SHA-384/512 uses type word64 (unsigned long long) in the internal state. And you deleted the "NO_SHA512" from the MBED settings? You are using the MBED block in settings.h with "#define CYASSL_MBED" uncommented? Setting SIZEOF_LONG_LONG to 8 should be enabling the typedef for word64 in types.h.

I went back to the linux client and concatenated the pem files like you suggested, and it ran into the ASN 155 error, but that might be expected as the httpbin.org server has a certificate which expires before the server cert.
The test code still fails on the error -10.
Are there specific locations in the test code which I can print/test to make sure each step is being completed properly?
I did print out the hash and expected hash values to compare the values, but they look completely unrelated:

Hash  vs  test output
56      CB
CC      0
79      75
1D      3F
50      45
F2      A3
F9      5E
C       8B
8F      B5
DA      A0
C0      3D
B0      69
F2      9A
3E      C6
67      50
37      7
53      27
95      2C
83      32
0       AB
BB      E
3D      DE
7       D1
4D      63
A       1A
2A      8B
9C      60
C5      5A
E6      43
FC      FF
A4      5B
8       ED
2C      80
2F      86
12      7
F       2B
E7      A1
D4      E7
97      CC
35      23
FF      58
BA      BA
3D      EC
DC      A1
AB      34
D4      C8
75      25
82      A7

Note: The output mismatch from post 7 was due to a printf() formatting I missed, and is not relevant for the issues at hand.

If I remove the Sha384, then the Sha512 fails immediately afterwards with the same -10 error.

Note: If I remove the SHA512 ans SHA384 includes then the tests run through just fine. I would guess it is an issue with the SHA512 code that is cross-used by the SHA384 code, causing both to fail.

The device is Little-Endian (I tried with both Big-Endian and Little-Endian declared for various test runs, and only the Little-Endian works, not to mention I know from the board specs that it is Little-Endian)

As for the signer error, I went back and messed around with the cert ordering when concatenating, and was still getting the 155 error.
Note: I don't have access to the server itself, it's merely a public server we use to test HTTP access on. So if the issue is with their server-side certificate setup, then there's not much we can do to fix it.

OK. I see what's going on here. When running the wolfSSL client connecting to that server, you only need to add the root CA, AddTrustExternalRootCA. The server is sending its certificate, and the signing chain, including a copy of the AddTrust. When processing the Certificate message, wolfSSL ends up ignoring the AddTrust certificate and then checking the next certificate, COMODO RSA Certification Authority. That one is successfully verified and added to the CA pool. The next certificate, COMODO RSA Domain Validation Secure Server CA, is checked and is rejected because the signature fails.

This happens because by default wolfSSL is using fast math which uses preallocated buffers for the big integers, and the size defaults to 4096 to allow for the 2048x2048 bit multiply needed for RSA. The certificate COMODO RSA Certification Authority has a 4096-bit key, so the following RSA signature check fails.

I configured my local wolfSSL build to set FP_MAX_BITS=8192 which will allow for the 4096x4096 bit multiply. (Actually, first I used non-fastmath, then fastmath with with the bigger max bits.) The connection succeeded.

This doesn't answer why the SHA-384 and SHA-512 hash tests fail for you since those are known-answer tests that don't involve RSA or the big integer math. (I'll follow up on that later.)

For places to print out data, you could add prints of the digest at the end of Sha384Update() and run it both on your device and on the desktop and see when they start to differ.

Which version of wolfSSL are you using right now and did you download it from our website?

Could you try

#define CTAOCRYPT_SLOW_WORD64

Here is a printout of all of the comiler warnings that appear:

Warning: Function "rotrFixed" was declared but never referenced in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/misc.c", Line: 70, Col: 26
Warning: Function "ByteReverseWords" was declared but never referenced in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/misc.c", Line: 97, Col: 20
Warning: Function "rotlFixed64" was declared but never referenced in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/misc.c", Line: 111, Col: 22
Warning: Function "rotrFixed64" was declared but never referenced in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/misc.c", Line: 117, Col: 22
Warning: Function "ByteReverseWords64" was declared but never referenced in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/misc.c", Line: 138, Col: 20
Warning: Function "xorbuf" was declared but never referenced in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/misc.c", Line: 159, Col: 20
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10155, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10156, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10157, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10158, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10173, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10174, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10175, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 10176, Col: 14
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11848, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11849, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11851, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11853, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11870, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11871, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11873, Col: 22
Warning: Assignment in condition in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/src/internal.c", Line: 11875, Col: 22
Warning: Function "printf" declared implicitly in "mtsas-dev/MTS-Socket-dev/HTTPClient-SSL/CyaSSL/ctaocrypt/src/sha512.c", Line: 405, Col: 5

Comparing the sha384 digest between the board and the linux build, the initialize functions are called on both, but on the first Transform384() operation, the Linux code gets a different digest than the mbed version.
Something with the actual digest function must be different.

Here's the code I put to print out the debug data at the end of Transform384():

    for(int i=0; i<8; i++)
    printf("digest %d %llX\r\n", i, sha384->digest[i]); //REMOVE
    
    return 0;

The code to print the digests at the end of the Sha384Update():

    //REMOVE FROM
    static int x=0;
    for(int i=0; i<(SHA512_DIGEST_SIZE / sizeof(word64)); i++) {
        printf("digest(0)\t%llX\n",sha384->digest[i]); //REMOVE
    } x++;
    //REMOVE TO
    return 0;

The printouts for the actual digest data returned:

Mbed Nucleo Board:
digest(0)       CBBB9D5DC1059ED8
digest(0)       629A292A367CD507
digest(0)       9159015A3070DD17
digest(0)       152FECD8F70E5939
digest(0)       67332667FFC00B31
digest(0)       8EB44A8768581511
digest(0)       DB0C2E0D64F98FA7
digest(0)       47B5481DBEFA4FA4
digest 0 56CC791D50F2F90C
digest 1 8FDAC0B0F23E6737
digest 2 53958300BB3D074D
digest 3 A2A9CC5E6FCA408
digest 4 2C2F120FE7D49735
digest 5 FFBA3DDCABD47582
digest 6 FA79EF93D2258E9A
digest 7 68C05211D2BC1F91

LINUX:
InitSha384 Called
digest(0)    CBBB9D5DC1059ED8
digest(0)    629A292A367CD507
digest(0)    9159015A3070DD17
digest(0)    152FECD8F70E5939
digest(0)    67332667FFC00B31
digest(0)    8EB44A8768581511
digest(0)    DB0C2E0D64F98FA7
digest(0)    47B5481DBEFA4FA4
InitSha384 Called
digest(0)    CBBB9D5DC1059ED8
digest(0)    629A292A367CD507
digest(0)    9159015A3070DD17
digest(0)    152FECD8F70E5939
digest(0)    67332667FFC00B31
digest(0)    8EB44A8768581511
digest(0)    DB0C2E0D64F98FA7
digest(0)    47B5481DBEFA4FA4
digest 0 23FCDB853C1B60CB
digest 1 BC2332F642B7A150
digest 2 A39326D7E9F38FB7
digest 3 DBBBCFBBB544C85C
digest 4 F5983FEBD1710751
digest 5 566BFD073BE1C356
digest 6 A07A863D879EDD50
digest 7 B56481C7230EB91D

Iteration 3: W[1] is the first different word during the hashing;
h(i) changes by 2-6 bits to cause it to change the W[1] value, which cascades to more wrong numbers later.
h(i) outputs: 9CB641F2CED773F3 (On the MBED board)
                  : 9CB941F2CED774B3 (On the LINUX VM)

re 19:
I think you are showing the LS 32-bits of the buffer words. Right before the Transform384 in Sha384Final using the data "abc", I get

(lldb) p/x sha384->buffer
(word64 [16]) $18 = {
  [0] = 0x6162638000000000
  [1] = 0x0000000000000000
  [2] = 0x0000000000000000
  [3] = 0x0000000000000000
  [4] = 0x0000000000000000
  [5] = 0x0000000000000000
  [6] = 0x0000000000000000
  [7] = 0x0000000000000000
  [8] = 0x0000000000000000
  [9] = 0x0000000000000000
  [10] = 0x0000000000000000
  [11] = 0x0000000000000000
  [12] = 0x0000000000000000
  [13] = 0x0000000000000000
  [14] = 0x0000000000000000
  [15] = 0x0000000000000018

This looks like "Chunk" 0 in the example you gave me from that external site. (I'm running on OS X.)

In 20, when you say iteration 3, are you talking about the loop in Transform384? (So, technically iteration 48, since that loop is doing 16 iterations at a time?)

Since your compiler is trying to do 64-bit stuff with a 32-bit compiler, maybe it is tripping up on not rotating something enough, or truncating a 64-value at 32-bits in the wrong spot.

I don't think it is accessing outside the tables with negative array indices because they are masked to 3 or 4 bits. (unless there are shenanigans regarding the types of the bare numbers.) And there shouldn't be any assumptions made about locals being initialized, beyond they aren't.

Yes, you are right, I keep forgetting to modify the printf() to display the 64-bit values properly. When I went back and displayed it, it is showing the data prefixed to the first W[0] properly.

Yes, when I meant iteration 3, I meant that it had run through the 16 times processing twice, and was then on the third run through the 16 times loop. So, in that case, W[1] is actually the 34th "iteration" of the hash (2*16 + 2).

I agree with you on the table indices. I saw the bit masking on it, but still wanted to point it out as a possible cause.
And if no initializations are assumed, the variable declarations should be fine as well.

I am going to see if unrolling the bit operations (printing them) will show up any issues, but with how it runs fine through the first 32 operations just fine, it seems odd that it wouldn't screw up on the first iteration.

@Post 23 (Tangent): The mbed compiler does parallel compiling (I think), which might explain why the warnings show up whether they should or not. And they aren't consistent tbh, as sometimes the compiler will compile with no problems.

@32 vs 64 bit-ops: I'll be going through the bit operations and checking that the compiler runs the code as we would expect. All of the intermediate T[0] arrays contain the correct values when running on either platform (at least, up until the first W[1] is wrong)