Chapter 25. Diagnostics
CloverDX Server allows you to create a thread dump or a heap dump for both Server Core and Worker. The heap and thread dumps are useful for investigation of performance and memory issues.
A heap dump is a content of the JVM process memory stored in a binary file.
The generated heap dump file can be investigated with tools like
A thread dump is a list of existing JVM threads with their callstacks and held locking objects (if supported). It can be viewed in a text editor.
Note that generating a heap or thread dump requires the Heap Memory Dump permission.
Downloading Heap/Thread Dump
In Server GUI, go to→ → .
To download a heap dump or thread dump of either Server Core or Worker, click on the respective link. Note that in the case of a heap dump, it may take a moment before the download is ready.
The temp file is automatically deleted after download, but note that it may contain sensitive information (e.g. passwords) in plain text.
A garbage collection is triggered before the heap dump which may cause the Server to briefly stop responding.
The Dump live objects only checkbox allows you to avoid dumping of objects awaiting garbage collection.
Downloading Heap/Thread Dump Using
The heap dump of Worker can be created with jcmd command:
jcmd <pidOfWorker> GC.heap_dump <filename>
You should specify the file name with full path to avoid searching for the file
jcmd does not always create it in the working directory.
The thread dump of Worker can be created with jcmd command:
jcmd <pidOfWorker> Thread.print
See details on jcmd.
Generating Heap Dump on Out of Memory Errors
To generate a heap dump on Out of Memory errors, add
-XX:+HeapDumpOnOutOfMemoryError to the
A dump file
java_pid.hprof will be generated when an Out of Memory error occurs.
The heap dump will be located in the working directory of the Worker's process
(same as the Server Core's working directory).
You can override the directory location with the
Important: the generated file can be large, its size is equal to the heap size.
Enabling GC Logging
Some memory and performance issues can be investigated with help of garbage collection logs.
To enable the logging of the Server Core, add
-verbose:gc -XX:+PrintGCTimeStamps -Xloggc:server_core_gc_log.txt to
Restart CloverDX Server for changes to take effect.
To enable the garbage collection logging in Worker, add the flags
-verbose:gc -XX:+PrintGCTimeStamps -Xloggc:worker_gc_log.txt
to the Worker's JVM arguments field in the Setup GUI,
or the worker.jvmOptions property.
-Xloggc option sets the path for the detailed garbage collector log.
For the GC logging of Worker, use a different file name than for the Server.
You can analyze the log file with various tools, e.g. http://gceasy.io/.
Restart Worker for changes to take effect.
Another useful garbage collector flags for Java 8 are:
For details on JVM flags, see Oracle's Java HotSpot VM Options.
GC Logs Overwriting Prevention
If GC logging is enabled and restart of Server Core or Worker is required, a new log is created which overwrites the old log containing important information. To prevent this, you can add a timestamp to the log's filename:
-verbose:gc -XX:+PrintGCTimeStamps -Xloggc:/data/logs/gc_worker_%t.log
You can change the
/data/logs/gc_worker path as required.
Also remember that to prevent overwriting the file, you should change the filename depending on whether you are changing the flag in Server Core or Worker.
For changes to take effect, restart CloverDX Server or Worker, if you changed the flag for Server Core or Worker, respectively.
Additional Diagnostic Tools
Below are additional useful diagnostic tools:
Investigating usage of direct memory
To investigate usage of direct memory,
the worker.jvmOptions property.
The details on native memory usage can be displayed with
jcmd <pid> VM.native_memory summary.
For more information, see Native Memory Tracking tool.
Enable Remote JMX Monitoring
Add the following options to the worker.jvmOptions property:
-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8687 -Dcom.sun.management.jmxremote.rmi.port=8687 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=example.com
With the above options, you enable remote connection to JMX monitoring,
which provides a wide range of information about the running JVM, JNDI resources, etc.
Change the value of
java.rmi.server.hostname to the hostname of your Server.