Version

    8. Postinstallation Configuration

    Memory Settings
    Maximum Number of Open Files
    Maximum Number of Processes or Threads
    Reflective access
    Firewall Exceptions
    Garbage Collector for Worker
    Reverse Proxy Configuration

    Memory Settings

    Current implementation of Java Virtual Machine allows only a global configuration of memory for the JVM system process. Thus the whole application server, together with WARs and EARs running on it, share one memory space.

    Default JVM memory settings is too low for running an application container with CloverDX Server. Some application servers increase JVM defaults themselves, however they still may be too low.

    The optimal memory limits depend on many conditions, i.e. transformations which CloverDX should execute. Please note that the maximum limit isn’t the amount of permanently allocated memory, but limit which can’t be exceeded. If the limit is exhausted, the OutOfMemoryError is raised.

    JVM Memory Areas

    JVM memory consists of several areas: heap memory, Metaspace, direct memory and stack memory. Since JVM memory is not just HEAP memory, you should not set the HEAP limit too high; in case it consumes whole RAM, JVM won’t be able to allocate direct memory and stack for new threads.

    Table 2. JVM Memory Structure
    Type Description

    Heap memory

    Heap is an area of memory used by JVM for dynamic memory allocation. Required heap memory size depends on various factors (e.g. complexity of graphs, number of graphs running in parallel, type of component, etc.), see the respective server container’s installation guide in this documentation. (Note that current heap memory usage can be observed in CloverDX Server Console.)

    Metaspace

    Separate memory space containing class definitions and related metadata.

    Direct Memory

    Memory used by buffers for I/O operations.

    Stack Memory

    Stack Memory contains local, method specific variables and references to other objects in the method. Each thread has its own stack; therefore, the memory usage depends on the number of components running in parallel.

    Configuring Memory

    You can set the minimum and maximum memory heap size by adjusting the "Xms" and "Xmx" JVM parameters. There are more ways to change the settings depending on the used application container.

    Recommended Server Core and Worker Heap Memory Configuration

    Optimal distribution of main memory between Server Core and Worker depends on the nature of executed tasks. The recommended defaults of Server Core heap size and Worker heap size for different RAM sizes are in the table below.

    Heap limit is not a limit of the full memory used by JVM. JVM uses memory in addition to the heap size for other memory spaces, e.g. direct memory. We recommend to set the heap limit to no more than 75% of system memory size, to leave space for the operating system and other JVM memory spaces.

    Table 3. Recommended Heap Memory
    RAM Size Server Core Heap Worker Heap Remaining for OS (estimated)

    4 GB

    1 GB

    1 GB

    1 GB

    8 GB

    2 GB

    3 GB

    2 GB

    16 GB

    4 GB

    7 GB

    3 GB

    32 GB

    7 GB

    15 GB

    5 GB

    64 GB

    8 GB

    40 GB

    8 GB

    128 GB

    8 GB

    95 GB

    16 GB

    Memory Configuration in Java

    In Java, the memory space for loading classes (so called "Metaspace") is separated from heap, and can be set by the JVM parameter -XX:MaxMetaspaceSize. The default maximum Metaspace size is unlimited.

    Please see the specific container section for details on memory settings.

    Metaspace

    We recommend you to put limit on metaspace memory. Add -XX:MaxMetaspaceSize=size to command line parameters of Server Core or Worker. Replace size with a suitably high limit. 512MB should be enough.

    See https://docs.oracle.com/en/java/javase/11/tools/java.html

    Direct memory

    To avoid excessive usage of direct memory, we add -Djdk.nio.maxCachedBufferSize=262144 to Worker’s command line. We recommend you to add this system property to the command line of Server Core, as well.

    In Apache Tomcat, add this system property to JAVA_OPTS environment variable, which is configured in bin/setenv.sh file.

    Codecache Size

    Some CloverDX Server installations can occasionally run into performance issue: JVM is running more than hundred times slower. The issue can be caused by a full code cache (Java SE Embedded: Developer’s Guide - Codecache Tuning). The reserved code cache size is platform dependent and can be too small for CloverDX Server. It is highly recommended to increase the code cache size using the following JVM argument:

    -XX:ReservedCodeCacheSize=256m

    Maximum Number of Open Files

    When using resource-demanding components, such as FastSort, or when running a large number of graphs concurrently, you may reach the system limit on simultaneously open files. This is usually indicated by the java.io.IOException: Too many open files exception.

    The default limit is fairly low in many Linux distributions (e.g. 4,096 in Ubuntu). Such a limit can be easily exceeded, considering that one FastSort component can open up to 1,000 files when sorting 10 million records. Furthermore, some application containers recommend increasing the limit themselves.

    Therefore, it is recommended to increase the limit for production systems. Reasonable limits vary from 10,000 to about 100,000 depending on the expected load of CloverDX Server and the complexity of your graphs.

    The current limit can be displayed in most UNIX-like systems using the ulimit -Sn command.

    The exact way of increasing the limit is OS-specific and is beyond the scope of this manual.

    Maximum Number of Processes or Threads

    If you run graphs with many subgraphs containing many components, you may reach the limit on number of threads per user or per system. In this case, you can find java.lang.OutOfMemoryError: unable to create new native thread in a graph’s log.

    The current limit on number of processes/threads per user can be displayed in most UNIX-like systems using the ulimit -Su command. Note that the documentation on ulimit may not distinguish between processes and threads. The limit on number of threads per system can be displayed using the sysctl kernel.threads-max command. The exact way of increasing the limit is OS-specific and is beyond the scope of this manual.

    Reflective access

    Some libraries require reflective access to function properly. Reflective access needs to be enabled by command line parameters.

    For Worker this is done automatically, but for Server Core this needs to be enabled manually by adding following parameters to command line of the application container.

    --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.xml/com.sun.org.apache.xerces.internal.dom=ALL-UNNAMED
--add-opens=java.base/javax.net.ssl=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.nio.file=ALL-UNNAMED
--add-opens=java.base/sun.nio.fs=ALL-UNNAMED
--add-opens=java.base/sun.security.util=ALL-UNNAMED

    Firewall Exceptions

    In order to function properly, CloverDX Server requires an outside communication. The table below describes both incoming and outgoing communication of CloverDX Server. Please, configure your firewall exceptions accordingly.

    Table 4. Firewall Exceptions
    Traffic Communication Description & Components

    Incoming

    HTTP(S)

    Communication between Designer and Server

    JMX

    Tracking and debugging information

    Outgoing (depending on an actual usage)

    JDBC

    Connection to databases (DatabaseReader, DatabaseWriter, DBExecute)

    JMS

    Receiving and sending JMS messages (JMSReader, JMSWriter, JMS Listener)

    HTTP(S)

    Requesting and receiving responses from servers (Readers, WebserviceClient, HTTPConnector)

    SMTP

    Sending data converted into emails (EmailSender)

    IMAP/POP3

    Receiving emails (EmailReader)

    FTP/SFTP:

    Remote file reading and writing (readers, writers)

    Garbage Collector for Worker

    We recommend using the G1 garbage collector for Worker. No configuration is necessary thanks to G1 being the default JVM garbage collector.

    If you want to configure the garbage collector yourself, you can do so using the worker.jvmOptions property.

    See also Enabling GC Logging.

    Reverse Proxy Configuration

    CloverDX Server instance can run behind a HTTP proxy that provides services such as logging, SSL termination, caching, load balancing, access control, etc. However, the proxy may break some parts of the CloverDX Sever console. For instance, the endpoint URL for Data Services will not show the public URL used in the web browser, but the internal URL of CloverDX Server instance, as seen by the proxy. That is because the Data Service endpoint URL is constructed from the incoming request and the request no longer comes from the client, but from the proxy.

    As a workaround, you may configure the proxy to override the URL using the following HTTP headers:

    X-Forwarded-Proto

    URL scheme (protocol), e.g. "https"

    X-Forwarded-Host

    the hostname (and optionally, the port), e.g. "my-proxy-host:80"

    X-Forwarded-Port

    the port number, overrides X-Forwarded-Host, e.g. "80"

    X-Forwarded-Prefix

    the context path including the leading slash, e.g. "/mycontext"

    The following snippets show configuration examples for commonly used proxy servers.

    Example 1. Apache HTTP Server Proxy Configuration
    # Requires mod_proxy module
<Location "/mycontext">
    # Enable proxy
    ProxyPass "http://cloverdx-svc:8080/clover"
    # Pass "Host" header value; alternatively, set X-Forwarded-Host
    ProxyPreserveHost On
    # Fix "Location" response headers in redirects
    ProxyPassReverse "http://cloverdx-svc:8080/clover"

    # Add "X-Forwarded-Proto" header when running as a SSL terminator
    #RequestHeader set X-Forwarded-Proto "https"

    ### Only necessary when changing the context path from "/clover" to something else:
    # Add "X-Forwarded-Prefix" header to override the context path
    RequestHeader set X-Forwarded-Prefix "/mycontext"
    # Fix path in "Set-Cookie" headers
    ProxyPassReverseCookiePath "/clover" "/mycontext"
</Location>
    Example 2. NGINX Proxy Configuration
    events {
}

http {

  server { # simple reverse-proxy
    listen       80;

    location /mycontext {
        # Enable proxy
        proxy_pass          http://cloverdx-svc:8080/clover;
        # Pass "Host" header value; alternatively, set X-Forwarded-Host
        proxy_set_header    Host $http_host;
        # Fix "Location" response headers in redirects
        proxy_redirect      http://$http_host/clover http://$http_host/mycontext;

        # Add "X-Forwarded-Proto" header when running as a SSL terminator
        #proxy_set_header   X-Forwarded-Proto https

        ### Only necessary when changing the context path from "/clover" to something else:
        # Override context path
        proxy_set_header    X-Forwarded-Prefix /mycontext;
        # Fix path in "Set-Cookie" headers
        proxy_cookie_path   /clover /mycontext;
    }

  }

}

    arrow Continue with: System Database Configuration