Chapter 25. Diagnostics

This chapter introduces you into diagnostic tools that help you localize performance and memory issues.

CloverDX Server allows you to create a thread dump or a heap dump. The thread and heap dumps are useful for investigation of performance and memory issues.

In server GUI, go to ConfigurationSystem InfoDiagnostics.

Heap Dump

A heap dump is a content of the JVM process memory stored in a binary file. Creating a heap dump requires a Heap Memory Dump permission.

Server Core

To download a heap dump of the Server Core, click the Download button under the Heap Dump section. Downloading the heap dump may take some time.

The Dump live objects only checkbox allows you to avoid dumping of objects awaiting garbage collection.

Worker

The heap dump of Worker can be created with jcmd command: jcmd <pidOfWorker> GC.heap_dump <filename> You should specify the file name with full path to avoid searching for the file as jcmd does not always create it in the working directory.

See details on jcmd.

To generate a heap dump on Out of Memory errors, add -XX:+HeapDumpOnOutOfMemoryError to the worker.jvmOptions property. A dump file java_pid.hprof will be generated when an Out of Memory error occurs. The heap dump will be located in the working directory of the Worker's process (same as the Server Core's working directory). You can override the directory location with the -XX:HeapDumpPath=/disk2/dumps option. Important: the generated file can be large, its size is equal to the heap size.

The generated heap dump file can be investigated with tools like jvisualvm or jhat.

Heap Dump Analysis

You can use jvisualvm or jhat to view and analyze the heap dump.

[Important]Important

A heap dump does not work on WebSphere. On WebSphere, you can create heap dumps using the administration console. For the instructions, see IBM Knowledge Center.

Thread Dump

A thread dump is a list of existing JVM threads with their callstacks and held locking objects (if supported). It can be viewed in a text editor. Creating a thread dump requires a Heap Memory Dump permission.

Server Core

To download the thread dump, click the Download button under the Thread Dump section.

Worker

The thread dump of Worker can be created with jcmd command: jcmd <pidOfWorker> Thread.print

See details on jcmd.

Enabling GC Logging

Some memory and performance issues can be investigated with help of garbage collection logs.

Core

To enable the logging of the Server Core, add -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:server_core_gc_log.txt to JAVA_OPTS in $CATALINA_HOME/bin/setenv.sh. Restart of the Server is required to reflect the configuration change.

Worker

To enable the garbage collection logging in Worker, add the flags -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:worker_gc_log.txt to the Worker's JVM arguments field in the Setup GUI, or the worker.jvmOptions property.

The -Xloggc option sets the path for the detailed garbage collector log. For the GC logging of Worker, use a different file name than for the Server. You can analyze the log file with various tools, e.g. http://gceasy.io/.

Restart of Worker is required.

More Details

Another useful garbage collector flags are -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:+PrintTenuringDistribution

For details on JVM flags, see Oracle's Java HotSpot VM Options.

Additional Diagnostic Tools

Below are additional useful diagnostic tools:

Investigating usage of direct memory

To investigate usage of direct memory, add -XX:NativeMemoryTracking=summary the worker.jvmOptions property. The details on native memory usage can be displayed with jcmd <pid> VM.native_memory summary. For more information, see Native Memory Tracking tool.

Enable remote JMX monitoring

Add the following options to the worker.jvmOptions property:

-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8687 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=example.com

With the above options, you enable remote connection to JMX monitoring, which provides a wide range of information about the running JVM, JNDI resources etc. Change the value of java.rmi.server.hostname to the hostname of your Server.