• Registered: 2011-07-28
  • Posts: 16

Topic: sniffer memory leak

I use the sniffer on a production environment.
I configured 1 host to be sniffed.
the sniffertest process crashes after a while after the process have consumed all system memory.

Share

  • Registered: 2011-07-28
  • Posts: 16

Re: sniffer memory leak

the problem seems to be related to non decoded packets
when all data is decoded the memory usage seems to not increase

Share

  • From: Bozeman, MT
  • Registered: 2010-10-20
  • Posts: 650

Re: sniffer memory leak

Hi Luca,

Can you tell us more about your setup, including:

- Are you running in a VM?  What OS and version of CyaSSL embedded SSL are you using?
- How many connections are you tracing?
- Are you noticing any missed packets or connection errors?
- How much RAM do you have available?

If you are missing packets, it may be possible that this is causing the sniffer connections to buffer lots of memory (thus the increase in memory you are seeing).

Thanks,
Chris

https://www.wolfssl.com

chrisc's Website

Share

  • Registered: 2011-07-28
  • Posts: 16

Re: sniffer memory leak

Hi chris
it runs on a ubuntu 11.04 server, kernel 2.6.38-13-server #57-Ubuntu SMP
I tried with official 2.3.0 and the latest from git
At the moment I cannot say how many socket connections
I know I see an average of 60-80k http requests per minute
RAM 8G
I need to verify if we miss packets.
If packet are missed they can be 2 reasons:
1) span/tap port saturation
2) ssl decoding not enough fast?

Can only be a problem related to missed packets? If a connection gives a lot of errors (because unknown cypher or the ssl compression) cannot cause the same problem?

There is a way to make it more reliable?
The process crashes each 40/50 sec and the sniffer is unusable.
There is an api for dump the buffer status?
Or maybe an api for flush connections not correctly decoded?

thanks,
Luca

Share

  • From: Bozeman, MT
  • Registered: 2010-10-20
  • Posts: 650

Re: sniffer memory leak

Hi Luca,

We just pushed a commit related to memory usage with the sniffer.  Can you give our most recent GitHub code a try and see if you are still seeing the same problem?

There is an api for dump the buffer status?
Or maybe an api for flush connections not correctly decoded?

We don't currently have these options available in our API, no.  We'll have to work on adding early flushing of bad sessions and a way to flag missed packets as fatal.

- Chris

https://www.wolfssl.com

chrisc's Website

Share

  • Registered: 2011-07-28
  • Posts: 16

Re: sniffer memory leak

Hi Chris,

thanks for the patch, tomorrow I can test it and I let you know if the problem is solved.

We'll have to work on adding early flushing of bad sessions and a way to flag missed packets as fatal.

perfect!

Luca

Share

  • Registered: 2011-07-28
  • Posts: 16

Re: sniffer memory leak

I get the following error by compiling latest from git:

  CC     ctaocrypt/src/src_libcyassl_la-sha512.lo
  CC     src/src_libcyassl_la-sniffer.lo
src/sniffer.c: In function 'GetSnifferSession':
src/sniffer.c:848:5: error: comparison of unsigned expression >= 0 is always true [-Werror=type-limits]
src/sniffer.c: In function 'RemoveSession':
src/sniffer.c:1588:5: error: comparison of unsigned expression >= 0 is always true [-Werror=type-limits]
cc1: all warnings being treated as errors

make[1]: *** [src/src_libcyassl_la-sniffer.lo] Error 1
make[1]: Leaving directory `/home/luca/dev/cyassl'
make: *** [all] Error 2

Luca

Share

  • todd
  • Moderator
  • Offline
  • Registered: 2010-10-21
  • Posts: 98

Re: sniffer memory leak

Just commited a fix.

You can always remove -Werror from the Makefile to get past these warning errors as we're testing all compilers/versions/systems with the recent decision to treats warnings as errors in the source tree.  The release versions won't have this restriction.

Share

  • Registered: 2011-07-28
  • Posts: 16

Re: sniffer memory leak

thanks Todd, there is still some other warnings, but I removed the -Werror from makefile.

it seems the memory leak is still here. I see a huge number of packets that cannot be decoded.

Share

  • Registered: 2011-07-28
  • Posts: 16

Re: sniffer memory leak

If helps, I don't see drops in the capture but they are a lot of retransmission

Share