Version

    Chapter 26. Diagnostics

    CloverDX Server allows you to create a thread dump or a heap dump for both Server Core and Worker. The heap and thread dumps are useful for investigation of performance and memory issues.

    A heap dump is a content of the JVM process memory stored in a binary file. The generated heap dump file can be investigated with tools like jvisualvm or jhat.

    A thread dump is a list of existing JVM threads with their callstacks and held locking objects (if supported). It can be viewed in a text editor.

    Note that generating a heap dump requires the Heap Memory Dump permission.

    Downloading Heap/Thread Dump

    In Server GUI, go to Configuration → System Info → Diagnostics.

    To download a heap dump or thread dump of either Server Core or Worker, click on the respective button. Note that in the case of a heap dump, it may take a moment before the download is ready.

    [Important]Important

    The size of the heap dump temp file is determined by the Server Core or Worker heap memory size setting - make sure there is enough free space in the temp directory before downloading the file.

    The temporary file is automatically deleted after download, but note that it may contain sensitive information (e.g. passwords) in plain text.

    A garbage collection is triggered before the heap dump which may cause the Server to briefly stop responding.

    The Dump live objects only checkbox allows you to avoid dumping of objects awaiting garbage collection.

    Downloading Heap/Thread Dump Using jcmd Command

    Heap Dump

    The heap dump of Worker can be created with jcmd command: jcmd <pidOfWorker> GC.heap_dump <filename> You should specify the file name with full path to avoid searching for the file as jcmd does not always create it in the working directory.

    Thread Dump

    The thread dump of Worker can be created with jcmd command: jcmd <pidOfWorker> Thread.print

    See details on jcmd.

    Generating Heap Dump on Out of Memory Errors

    To generate a heap dump on Out of Memory errors, add -XX:+HeapDumpOnOutOfMemoryError to the worker.jvmOptions property. A dump file java_pid.hprof will be generated when an Out of Memory error occurs. The heap dump will be located in the working directory of the Worker's process (same as the Server Core's working directory). You can override the directory location with the -XX:HeapDumpPath=/disk2/dumps option. Important: the generated file can be large, its size is equal to the heap size.

    Garbage Collection Logging

    Some memory and performance issues can be investigated with help of garbage collection (GC) logs. By default, CloverDX Server enables Worker's GC logging automatically, but the user can override these default settings, as required.

    GC log files are stored in temp/cloverlogs/gc. The name of the file contains Worker and node identification, and a timestamp which prevents log overwriting, for example: worker0_node1-gc-2019-07-18_15-18-58.log. When a Worker process starts, it creates a new file name with the current timestamp. While the Worker process runs, files with this name are rotated. Log rotation amount is 10 files with the maximum size of 5MB per file.

    You can analyze the log file with various tools, e.g. http://gceasy.io/.

    Tweaking Garbage Collection Logging

    GC Logging can be set up manually for both Server Core and Worker using the flags listed below. Note that if you overwrite even one of the flags for Worker, CloverDX Server default GC logging setting is ignored. You can change the name and path to the file (include the timestamp in the file name to prevent overwriting):

    • Java 8: -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:/data/logs/gc_worker_%t.log

    • Java 11: -Xlog:gc*:file=/data/logs/gc_worker_%t.log:time:filecount=10,filesize=5M

    • IBM SDK Java 8: -verbose:gc -Xverbosegclog:/data/logs/gc_worker_%Y-%m-%d_%H-%M-%S.log

    The flags should be added to:

    Restart CloverDX Server for changes to take effect.

    More Details

    Another useful garbage collector flags for Java 8 are: -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:+PrintTenuringDistribution

    For details on JVM flags, see Oracle's Java HotSpot VM Options.

    Performance Logging

    CloverDX regularly collects performance metrics and stores them in the performance log. The metrics are such as CPU load, garbage collector activity, used heap, thread counts etc. The performance log is an additional tool to analyze an incident, see Performance Log for more details.

    Additional Diagnostic Tools

    Below are additional useful diagnostic tools:

    Investigating usage of direct memory

    To investigate usage of direct memory, add -XX:NativeMemoryTracking=summary to the worker.jvmOptions property. The details on native memory usage can be displayed with jcmd <pid> VM.native_memory summary. For more information, see Native Memory Tracking tool.

    Enable Remote JMX Monitoring

    Add the following options to the worker.jvmOptions property:

    -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8687 -Dcom.sun.management.jmxremote.rmi.port=8687 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=example.com

    With the above options, you enable remote connection to JMX monitoring, which provides a wide range of information about the running JVM, JNDI resources, etc. Change the value of java.rmi.server.hostname to the hostname of your Server.