Building jemalloc is outside the scope of this article and should be a trivial process. However, it is important to ensure that the jemalloc binary being used includes profiling support. I'll be assuming that jemalloc is installed in this path: /opt/jemalloc, this should have the bin and lib directories under it.
The Getting Started page on the jemalloc wiki talks about some of the environment variables that need to be set up for jemalloc to work. We'll get back to the specifics of this because first we need to have a way of telling WebLogic about these environment variables. That is done via the use of the setUserOverrides.sh file as per the documentation...
To enable you to customize server startup parameters that apply to all servers in a domain, you can create a file called setUserOverrides.cmd (Windows) or setUserOverrides.sh (UNIX) and configure it to, for example, add custom libraries to the WebLogic Server classpath, specify additional java command line options for running the servers, or specify additional environment variables. Any customizations you add to this file are preserved during domain upgrade operations, and are carried over to remote servers when using the pack and unpack commands.
During server startup, if this file exists, it is included in the startup sequence and any overrides it defines take effect. The file must be stored in the domain_home/bin directory.
During server startup, if this file exists, it is included in the startup sequence and any overrides it defines take effect. The file must be stored in the domain_home/bin directory.
So after creating this file, we can add the required variables to it to load and configure jemalloc...
$DOMAIN_HOME/bin/setUserOverrides.sh
export LD_PRELOAD="/opt/jemalloc/lib/libjemalloc.so"
export MALLOC_CONF="abort_conf:true,prof:true,lg_prof_interval:30,lg_prof_sample:19,prof_prefix:jeprof"
That's actually all there is to it! After restarting WebLogic, it will use jemalloc as its memory allocator and all of the heap profile data will be captured and dumped into the $DOMAIN_HOME directory.
However we're not quite done. What do those configuration parameters actually mean? Lets look at each one in more detail. These are all described on the jemalloc man page.
abort_conf:true
opt.abort_conf (bool) r-
Abort-on-invalid-configuration enabled/disabled. If true, invalid runtime options are fatal. The process will call abort(3) in these cases. This option is disabled by default unless --enable-debug is specified during configuration, in which case it is enabled by default.
Abort-on-invalid-configuration enabled/disabled. If true, invalid runtime options are fatal. The process will call abort(3) in these cases. This option is disabled by default unless --enable-debug is specified during configuration, in which case it is enabled by default.
By setting abort_conf to true we make sure that all of the configuration parameters that are specified are valid. This is a nice way of making sure we don't have bad configuration parameters as WebLogic will be forced to terminate on startup.
prof:true
opt.prof (bool) r- [--enable-prof]
Memory profiling enabled/disabled. If enabled, profile memory allocation activity. See the opt.prof_active option for on-the-fly activation/deactivation. See the opt.lg_prof_sample option for probabilistic sampling control. See the opt.prof_accum option for control of cumulative sample reporting. See the opt.lg_prof_interval option for information on interval-triggered profile dumping, the opt.prof_gdump option for information on high-water-triggered profile dumping, and the opt.prof_final option for final profile dumping. Profile output is compatible with the jeprof command, which is based on the pprof that is developed as part of the gperftools package. See HEAP PROFILE FORMAT for heap profile format documentation.
Memory profiling enabled/disabled. If enabled, profile memory allocation activity. See the opt.prof_active option for on-the-fly activation/deactivation. See the opt.lg_prof_sample option for probabilistic sampling control. See the opt.prof_accum option for control of cumulative sample reporting. See the opt.lg_prof_interval option for information on interval-triggered profile dumping, the opt.prof_gdump option for information on high-water-triggered profile dumping, and the opt.prof_final option for final profile dumping. Profile output is compatible with the jeprof command, which is based on the pprof that is developed as part of the gperftools package. See HEAP PROFILE FORMAT for heap profile format documentation.
Setting prof to true enables memory profiling.
lg_prof_interval:30
opt.lg_prof_interval (ssize_t) r- [--enable-prof]
Average interval (log base 2) between memory profile dumps, as measured in bytes of allocation activity. The actual interval between dumps may be sporadic because decentralized allocation counters are used to avoid synchronization bottlenecks. Profiles are dumped to files named according to the pattern <prefix>.<pid>.<seq>.i<iseq>.heap, where <prefix> is controlled by the opt.prof_prefix option. By default, interval-triggered profile dumping is disabled (encoded as -1).
Average interval (log base 2) between memory profile dumps, as measured in bytes of allocation activity. The actual interval between dumps may be sporadic because decentralized allocation counters are used to avoid synchronization bottlenecks. Profiles are dumped to files named according to the pattern <prefix>.<pid>.<seq>.i<iseq>.heap, where <prefix> is controlled by the opt.prof_prefix option. By default, interval-triggered profile dumping is disabled (encoded as -1).
This sets how often a heap profile dump file will be written to disk, measured by the number of bytes allocated. The number is specified as a power of 2 so by setting this to 30 it means that every 2^30 bytes (on average) we will get a heap profile dump file. In easier to understand terms...2^30 bytes = 1,073,741,824 bytes, which is 1,048,576 kilobytes, which is 1,024 megabytes, which is 1 gigabyte. So after the process allocates 1 gigabyte of memory, a profile dump file is written. This seems like a decent interval to use for an application server.
lg_prof_sample:19
opt.lg_prof_sample (size_t) r- [--enable-prof]
Average interval (log base 2) between allocation samples, as measured in bytes of allocation activity. Increasing the sampling interval decreases profile fidelity, but also decreases the computational overhead. The default sample interval is 512 KiB (2^19 B).
Average interval (log base 2) between allocation samples, as measured in bytes of allocation activity. Increasing the sampling interval decreases profile fidelity, but also decreases the computational overhead. The default sample interval is 512 KiB (2^19 B).
This sets how often jemalloc will take a sample of memory allocation, again measured in terms of average number of bytes allocated. The default value of 19 is used here so we don't really need to set it. Again this is expressed as a power of 2, so 2^19 bytes i.e. 524,288 bytes, 512 kilobytes.
prof_prefix:jeprof
opt.prof_prefix (const char *) r- [--enable-prof]
Filename prefix for profile dumps. If the prefix is set to the empty string, no automatic dumps will occur; this is primarily useful for disabling the automatic final heap dump (which also disables leak reporting, if enabled). The default prefix is jeprof.
Filename prefix for profile dumps. If the prefix is set to the empty string, no automatic dumps will occur; this is primarily useful for disabling the automatic final heap dump (which also disables leak reporting, if enabled). The default prefix is jeprof.
The prefix value to use for the name of the profile dump file. This can be set to a path name so if we want to output these dump files somewhere other than the $DOMAIN_HOME, we can set it to that path e.g. /tmp. Of course the path has to be writeable. In this case the prefix is set to the default value, jeprof.
Other configuration parameters can of course be set but I did't find them of any value. For example, prof_accum is not very useful since it requires additional memory and cumulative counts for a long running process are of no real value i.e. we know memory allocation will keep increasing. The above configuration gives memory usage values at an instant in time, which can be used to see patterns of memory utilisation. The prof_final option is also of no use since we don't expect that WebLogic would be be shut down just to get the profile stats. The above configuration dumps these stats periodically. Other configuration parameters are similarly of not much use in this case.
I know that I talked about kilobytes, megabytes and gigabytes above, technically this should have been kibibytes, mebibytes and gibityes but I think most people are familiar with their incorrectly canonised terms, so I stuck to those.
In a future article I'll talk about how jemalloc profile dumps can be analysed to show potential memory leaks over a time period. Keep an eye out for that article.
-i
