If you have or suspect you have a memory leak in your application, you simply cannot go past using jemalloc to help you track it down. It's an implementation of malloc() and related functions that provide all the usual memory allocation/deallocation functionality as well as profiling and heap allocation stats capture. These can be invaluable in helping track down memory leaks, but there are some caveats. Documentation is not perfect and some of the output formats are hard to understand and reuse for statistical analysis.

There is documentation in the jemalloc Git repository that talks about leak checking. I'm not going to go into details on how to set up jemalloc here and will assume that it has already been set up and the heap dump profile files are already available (you can read about how to set it up here). There are essentially two ways to analyse these files, both are done with the aid of the jeprof tool. The first is generating a text representation of memory usage and second is to generate a graph that shows both memory usage and function call relationships. For statistical analysis the former is more appropriate since we can use that information over many profile dumps to plot graphs showing memory usage over time.

For a single profile dump, we can run jeprof to generate our required output like this...


...which produces output that is very human readable and nice but is not ideal for further processing...


Since jemalloc aligns to gperftools to some extent, the format of the above output is as per the gperftools documentation described here.




In terms of statistical analysis, the percentage columns do not really matter. Then, the column that has the procedure name is the last column, which isn't very spreadsheet friendly. This output is good for manual debugging of a single heap dump, but if you want to do further analysis and comparison to other dumps the only columns that are of interest are 1, 4 and 6 i.e direct memory use, procedure and callee memory use and the procedure name/address.

To capture data that is of interest then requires some basic filtering with grep and awk...


The use of grep ensures that we only get lines that have actual data (looking for %age values), ignoring any other lines like total counts, etc. Then awk is used to rearrange and extract data from columns 1, 4 and 6. The output from that on the same profile dump as earlier looks like this...


That's not as human readable but is much more machine friendly!

Of course looking at one profile dump is not enough to see trends. Since each dump file has memory usage statistics at that instance in time, it is necessary to look at multiple dumps to get an understanding of the behaviour of procedures showing where memory is being allocated and potentially leaked.

I used a simple Bash script to process all of the jeprof dump files into the more friendly machine readable format and then went on to processing those files further and plotting memory usage trends. These are topics for another blog post, which I'll link to here once they are available.

-i