Overview of Testing in wolfSSL

    • The security of wolfSSL products is always on our mind and holds high importance.  Conducting regular, diligent, and well-planned testing helps maintain wolfSSL’s robustness and security.  We strive to write and maintain clean, readable, and understandable code.

    Like the halting problem, we know it is impossible to test every single possible path through the software, but we practice an approach that is focused on lowering risk of failure. In addition to extensive automated testing, we make sure that we specifically test well-known use cases. This post outlines some of our internal testing process.

    1. API Unit Testing:  We have unit tests in place that test API functions for correct behavior. This helps maintain library consistency across releases and as the code evolves.  It helps us to deliver a high quality well tested API to our end users with each software release.  API unit tests are run with each “make check” of wolfSSL.

    1. Cipher Suite Testing: wolfSSL supports an extensive list of cipher suites, which are all tested with every “make check” using the wolfSSL example client and example server.  Each cipher suite is tested not only in the default configuration, but also in non-blocking mode and with client authentication both turned on/off.

    1. Algorithm Testing: The security of our SSL/TLS implementation depends on the correctness and robustness of our underlying cryptography library, wolfCrypt.  We test all algorithms using NIST test vectors in addition to running our CAVP test harness used for our FIPS 140-2 validations.  We also test on both big and little endian platforms for portability.

    1. Benchmark Testing: We engage in another ever expanding universe of benchmark testing, where we look at sizing, transmission rates, connection speeds, and cryptography performance.  A version of our benchmark suite is included in every download for users to enjoy!

    1. Static Analysis: We do static analysis on our entire codebase using not only one, but multiple different static analysis tools.  We currently use Coverity Scan, clang scan-build, and Facebook infer.  These tools help us to automatically find bugs including ones on low-traffic code paths.

    1. Detecting Memory Errors:  We mitigate memory errors by using valgrind on a regular and automated basis.  This helps find memory errors including invalid access, use of undefined values, incorrect freeing of dynamic memory, and memory leaks.

    1. Interop Testing: We test for interoperability with other Open Source TLS implementations, including OpenSSL, BoringSSL, and GnuTLS.  This helps us to catch any protocol implementation errors in either wolfSSL or the implementation being tested against.  We also test outside of a closed environment by connecting to servers in the real world running unknown SSL/TLS implementations.

    1. Real World Builds: We build with a series of ‘real’ applications, like cURL, wget, pppd, OpenSSH, stunnel, lighttpd, etc.  For some of our customers with top level support, we build new releases with their application.

    1. Compiler Testing: We have users who compile wolfSSL with a variety of different compilers.  As such, we test compiling wolfSSL with many different compilers and toolchains including gcc/g++, clang, icc, Visual Studio, CodeWarrior, KDS, LPCXpresso, MPLAB XC, TI CCS, Keil, IAR, Cygwin, MinGW, CrossWorks, Arduino, Wind River Workbench, and more.

    1. Peer Review: More eyes on a codebase reduces bugs that end up in a final product.  Internally, we operate using a “Fork and Pull Request” model.  This means that every commit that makes it into our master branch has been reviewed and tested by at least two separate engineers.

    1. Third Party Testing: Our code is regularly reviewed by university researchers, customer and user security teams, FIPS and certification labs, and our Open Source user base.  This helps put more eyes on our code and product architecture.

    1. Fuzz Testing: We test using several different software fuzzers, including an in-memory fuzzer, a network fuzzer, OSS-fuzz, libfuzzer, tlsfuzzer, and AFL.  Fuzz testing bombards the program with invalid, unexpected, and random data that then allows for observing if there is potential memory leaks or logic errors.  This allows us to catch bugs that could turn into potential vulnerabilities if released in a final release.

    1. Continuous Integration (CI): Leveraging Jenkins, we run tests on each commit submitted to the wolfSSL code repository.  Tests run on each commit include testing of our FIPS build, numerous build options (customer/user/common), running valgrind, and doing static analysis with scan-build.

    1. Nightly Test Cycle: Each night we run extended tests that last longer than the typical ones during the work day.  These are more in-depth than our CI testing and puts results in our engineers’ inboxes each morning.  Some tests included in our nightly cycle include extended build option testing on multiple platforms with multiple compilers, and extended fuzz testing.

    If you have specific questions about how we test, please contact us at facts@wolfssl.com.  If you would like us to include your SSL/TLS or crypto implementation in our interop testing, please let us know!  Likewise, if you would like to include wolfSSL in your own test framework, we would be happy to discuss.