Version

    13. CentOS/RHEL RPM package

    Overview

    Starting with version 5.16, CloverDX Server is available for installation directly from our public repository. For convenience, the repository contains all the necessary resources to install CloverDX Server on CentOS 7 and Red Hat Enterprise Linux 7 Linux distributions with default, non-production configuration, complementing AWS Marketplace and Azure Marketplace listings.

    The benefit of installing CloverDX using this package repository is leveraging your system package manager’s benefits. You’ll always know when new version or update is available and may choose to install it (or not) along with other system updates during planned system downtimes.

    Prerequisites

    To be able to use the repository, download its YUM/DNF-compatible configuration file from https://packages.cloverdx.com/rpm/centos/7/cloverdx.repo or Customer Portal to /etc/yum.repos.d.

    curl -o /etc/yum.repos.d/cloverdx.repo https://packages.cloverdx.com/rpm/centos/7/cloverdx.repo

    Optionally also install the public GPG key, using rpm --import https://packages.cloverdx.com/rpm/pubkey/RPM-GPG-KEY-cloverdx used to sign all packages contained in our repository. Without this step, you’ll be prompted to trust the package vendor when installing the very first package.

    Architecture

    At this moment, the repository contains the following packages:

    1. Apache Tomcat 9 Application Server

    2. CloverDX Server (latest)

      • Base application server binaries

      • HTML documentation

      • Examples

      • Management UI

    3. AdoptOpenJDK repository, which installs the official Adoptium (formerly AdoptOpenJDK) configuration file into your /etc/yum.repos.d to satisfy all the dependencies set by CloverDX Server.

    Apache Tomcat 9

    Contains sources available from the official download site. The contents are unpacked into locations as proposed by LSB Workgroup, expecting OS to be managed by SystemD. It is possible to install this package on OS running SysV or other process managers but these would require additional installation steps, outlined in Operating systems without SystemD.

    Package installation directories:
    • /etc/tomcat9 (configuration files)

    • /usr/share/doc/tomcat9 (license and readme files)

    • /usr/share/tomcat9 (scripts and server’s binaries); CATALINA_HOME

    • /var/lib/tomcat9 (webapps, additional libraries); CATALINA_BASE

    • /var/cache/tomcat9 (work directory)

    • /var/log/tomcat9 (log files)

    • /var/tmp/tomcat9 (temporary files)

    All temporary file directories are managed by SystemD (cache, log, tmp). It is possible to adjust file retention periods and other properties using configuration file /usr/lib/tmpfiles.d/tomcat9.conf. Installator also creates new system user tomcat and makes all package directories under /var/{lib,cache,log,tmp}/tomcat9 owned by this user.

    SystemD configuration files:
    • /usr/lib/tmpfiles.d/tomcat9.conf

    • /usr/lib/sysusers.d/tomcat9.conf (only supported by some operating systems)

    • /usr/share/systemd/system/tomcat9.service

      It is imperative, /etc/tomcat9/conf.d/tomcat.conf contains at the very least CATALINA_HOME and CATALINA_BASE envvars. When not correctly configured, service will fail to start!

    If you’re running on an OS with other process manager than SystemD, please refer to Operating systems without SystemD.

    Configuration

    All Tomcat 9 configuration files are located in /etc/tomcat9. Package includes default configuration from original distribution but also adds blank setenv.sh file (linked to $CATALINA_BASE/bin) for convenience located in /etc/tomcat9 as its purpose is to set environment variables and other configuration. To get more detailed configuration information, refer to official Tomcat 9 documentation or install tomcat9-docs package and check http://<server-name>:8080/docs or /usr/share/tomcat9-docs/index.html to access offline documentation.

    Addons

    Repository also includes following optional Apache Tomcat 9 packages:

    • tomcat9-admin

    • tomcat9-docs

    • tomcat9-examples

    These packages contain webapps deployed directly to $CATALINA_BASE/webapps and are part of binary from official download site but not required to run application server itself.

    CloverDX Server

    Installed as separate instance of Apache Tomcat 9 (i.e. CATALINA_BASE) which shares application server’s binaries and startup scripts but uses own configuration. All configuration files are taken directly from Apache Tomcat package, changing HTTP port to 8083 to reflect default port on other out-of-the-box CloverDX installations.

    This change is not enough to encounter port collisions when parallel server is started. For example port 8009 (server management) remains unchanged. Therefore, is you’re running multiple parallel instances you may need to review /etc/cloverdx/server.xml.

    Package installation directories:

    Directories are mostly derived from requirement specified in Apache Tomcat’s "Advanced configuration" readme. All, but /etc/cloverdx and /usr/share/cloverdx are requirement of CATALINA_BASE directory.

    • /etc/cloverdx (configuration files)

    • /usr/share/cloverdx (CloverDX Server Web ARchive only)

    • /var/lib/cloverdx (webapps, additional libraries); CATALINA_BASE

    • /var/cache/cloverdx (work directory)

    • /var/log/cloverdx (log files)

    • /var/tmp/cloverdx (temporary files)

    SystemD configuration files:

    Similar to ones mentioned for Tomcat installation, same purpose, same locations just tomcat9 is substituted by cloverdx and instead of using /etc/tomcat9/conf.d/tomcat.conf, CloverDX uses /etc/cloverdx/clover.env with same content and purpose.

    Installation

    The CloverDX repository contains an installation package which adds an Adoptium repository to satisfy the Server’s dependency on its Java distribution. The following command therefore causes Adoptium’s Temurin JDK to be always up-to-date along with CloverDX Server.

    yum install -y adoptopenjdk-repo && yum install -y cloverdx

    adoptopenjdk-repo is programmatic way of following CentOS/RHEL instructions from Adoptium Docs and therefore if you choose to install Adoptium repository manually, following these instructions is an equivalent approach.

    Method 2: Separate Java installation

    If full and explicit control over the Java version is required, you may install Eclipse Temurin 17 JDK separately. CloverDX package has hard requisite on package temurin-17-jdk which may be downloaded from packages.adoptium.net or alternatively from other mirror. Such package may be installed either using rpm -i /path/to/downloaded/temurin-17-jdk.rpm or yum localinstall /path/to/downloaded/temurin-17-jdk.rpm.

    If this method is chosen, make sure to install JDK (not JRE).

    Installation of Java without use of package manager (rpm, yum or dnf) will result in Java not being recognized as satisfied dependency, resulting in failed installation of CloverDX Server. These unsupported installation methods include (but are not limited to) unzipping archive into destination directory (even standard one) or using bundled installers.

    When this pre-requisite is fulfilled, install CloverDX Server and Apache Tomcat 9 running yum install cloverdx.

    Post-installation checklist

    1. Configure /etc/cloverdx/clover.properties

    2. Create and put valid license into /etc/cloverdx/license.dat

    3. Obtain an SSL certificate and create JKS file using Java’s keytool

    4. Configure JMX with SSL

    5. Configure HTTPS for Tomcat in /etc/cloverdx/server.xml

    6. Configure SSL properties for Worker

    7. Open port 8083 (if you skipped SSL steps) or 8443 (if you configured default SSL port) on your firewall

    8. Start server using systemctl start cloverdx and test connection:

    You need to give server a couple of seconds before it starts up. When it is up and running, you should be able to receive "OK" as a response of the call.

    Configuration

    CloverDX Server is configured via configuration properties. See List of Configuration Properties for available configuration properties.

    Some of these properties are pre-defined in /etc/cloverdx/clover.properties.

    For security reasons and following best practices, this file’s permissions are set to read-only for any other user than root. This causes server UI to report an error that configuration file is unwriteable.

    This poses no limitations or issues other than being unable to change configuration directly from a web console. Depending on your organization’s policies, you may change owner of this file to clover:clover to allow configuration changes from the console.

    Most of instance’s configuration is set up by various scripts located in /etc/cloverdx/conf.d. These scripts are being loaded as part of Tomcat’s setenv.sh execution and are prefixed by order in which they should be loaded in. For example, files with 00_ prefix are loaded before ones with 10_ prefix.

    Default /etc/cloverdx/conf.d files are:

    • 00_java.sh - sets various Java-related attributes, to fine-tune process configuration for better performance

    • 00_memory.sh - based on total available memory, calculates reasonable heap sizes for core and worker; these are later used in 10_clover.sh and /etc/cloverdx/clover.properties; autoscaling can be disabled (see Memory)

    • 10_clover.sh - sets CATALINA_OPTS related to CloverDX server (configuration file locations etc.)

    • 10_gc.sh - Java Garbage collection settings

    • 10_jmx.sh - JMX monitoring related properties, see JMX monitoring

    • 99_derby.sh - Derby database data directory configuration;

      Delete this file on production environments or comment out its content! Derby database should never be used in production and is considered unsupported engine outside of use for CloverDX evaluation.

    Same purpose /etc/tomcat9/conf.d/tomcat.conf plays /etc/cloverdx/clover.env. This file always need to specify correct CATALINA_HOME and CATALINA_BASE directory. Other variables are optional but placed in the file for convenience.

    System Database Configuration

    Initial installation sets up a Derby database to persist data into /var/lib/cloverdx/derby (path is configured in /etc/cloverdx/conf.d/99_derby.sh). This database is not recommended for production use and should not be used in production due to low performance, lack of many features and is known for making upgrades unrealiable. This combination is therefore unsupported and only meant for evaluation purposes.

    To configure backend database into production setting:

    1. Get JDBC driver for your database and put into $CATALINA_BASE/lib (default: /var/lib/cloverdx/lib)

    2. Configure /etc/cloverdx/clover.properties according to general rules from server DB configuration section.

    3. For use with JNDI, it is also necessary to put connection details into /etc/cloverdx/Catalina/localhost/clover.xml, either following general instructions from Apache Tomcat JNDI resource manual or modify example from example for Apache Tomcat and PostgreSQL

    Libraries and Classpath

    Libraries are added to the classpath of Tomcat (i.e. Server Core) and Worker from directories under /var/lib/cloverdx:

    • lib/ - libraries to add to Tomcat and Server Core classpath (e.g. JDBC drivers)

    • worker-lib/ - libraries to add to Worker classpath (e.g. libraries used by jobs)

    To be able to use external library, just copy it into one (or both) of these directories. No other action is necessary.

    Sandboxes

    Sandboxes are stored by default in /var/lib/cloverdx/sandboxes directory. This can be changed using configuration file /etc/cloverdx/clover.properties.

    Memory

    Important memory settings are Java heap size for Server Core, Java heap size for Worker and sizes of additional Java memory spaces.

    The memory settings are automatically calculated based on the memory available to your machine. If you change the memory size, the heap memory settings are updated automatically during next start. For example if you modify the VM to use an instance type with a larger amount of memory, the heap sizes for Server Core and Worker will increase automatically.

    Formula is adjusted based on total available memory but for example; when running the VM with 16 GB of RAM, the Server Core will have 4 GB heap, Worker will have 7 GB heap, and the rest is left for additional Java memory spaces and the OS.

    Automatic memory settings can be suppressed by setting BOTH environment properties, via e.g. /etc/cloverdx/clover.env file or system .profile:

    • CLOVER_SERVER_HEAP_SIZE - heap size of Server Core (in MB)

    • CLOVER_WORKER_HEAP_SIZE - heap size of Worker (in MB)

    License

    The server can be activated via the Server Console, on the login screen or in Setup section after login.

    Default search location for license file is: /etc/cloverdx/license.dat, configured in /etc/cloverdx/clover.properties.

    CloverDX Services

    The CloverDX Server may be started automatically as a systemd service on startup. Simply add it to OS boot sequence by issuing sudo systemctl enable cloverdx. To manually stop/start/restart the server, use the following commands:

    • sudo systemctl stop cloverdx

    • sudo systemctl start cloverdx

    • sudo systemctl restart cloverdx

    To see the stdout logs of the clover service, use journalctl -u cloverdx | less. For working with the service, root permissions are required, use sudo to obtain them.

    When not running on an OS with SystemD, refer to chapter Operating systems without SystemD.

    Server logs

    All logs are collected to /var/log/cloverdx and its subdirectories. To change logging location, depending on its purpose; change:

    • /etc/cloverdx/logging.properties (all Apache Tomcat logs)

      • System log

      • Access log

    • /etc/cloverdx/logging-server.xml (log4j2 configuration file for CloverDX Server)

    • /etc/cloverdx/logging-worker.xml (log4j2 configuration file for Worker process)

    • /etc/cloverdx/clover.properties (cloverlogs.dir and logging_subdir for GC logs)

    For more detailed information about logging, refer to Logging chapter of this documentation.

    JMX monitoring

    Both Server Core and Worker are configured for JMX monitoring using tools like VisualVM or JConsole. The JMX is may be configured to use SSL (see SSL for JMX monitoring). It needs 2 open ports per process, so 4 ports in total. Ports are usually NOT allowed on firewalls, therefore, don’t forget to allow these (TCP) connections on your system firewall. To be able to connect, simply allow inbound connections from your IP to destination port range: 8686-8689.

    To connect through JMX, use port 8686 for Server Core, and port 8688 for Worker. Under some circumstances, system property java.rmi.server.hostname may need to be configured for JMX to run properly (see Appendix B of Oracle documentation).

    In case you want to customize the default JMX configuration, change:

    • Worker: /etc/cloverdx/clover.properties (worker.jvmOptions)

    • Core: /etc/cloverdx/conf.d/10_jmx.sh

    • Both: /etc/cloverdx/jmx.properties

    In most cases, you’d only need to modify /etc/cloverdx/jmx.properties. Process-specific files you should only modify when changing port assignments or when property requires to be assigned as system property.

    SSL for JMX monitoring

    Both Server (Core) and Worker are already pre-configured to be able to open SSL sockets. To enable this capability, it is necessary to obtain an SSL certificate and add into JKS using Java’s keytool utility. Step-by-step guide to create JKS storage is summarized in Oracle documentation.

    When JKS is created, uncomment all properties under Settings JMX with SSL of /etc/cloverdx/jmx.conf and configure appropriately javax.net.ssl.keyStore and javax.net.ssl.keyStorePassword. Then comment out (or remove) properties under Settings JMX without SSL.

    Upgrading CloverDX

    To upgrade, use your OS package manager, running yum update cloverdx (CentOS 7) or dnf update cloverdx (CentOS 8). This will replace old /usr/share/cloverdx/clover.war and installs new configuration under /etc/cloverdx. In case you made modifications to configuration, files with suffix .rpmsave containing your old settings will appear.

    When upgrade is done, Tomcat may detect change and re-deploy automatically but sometimes it does not. Running systemctl restart cloverdx is adviseable.

    It is recommended to take database backup before upgrade. This is because first boot may update its schema which could make database incompatible with previous version, making rollback more challenging. General guidelines as described in Server upgrade apply.

    To rollback an upgrade, run yum downgrade cloverdx (CentOS 7) or dnf downgrade cloverdx (CentOS 8) and restore database backup.

    Operating systems without SystemD

    Default installer relies on SystemD services in following areas:

    • Start/Restart/Stop and monitoring of Tomcat server process

    • Process dependency management (most notably - networking)

    • Environment initialization

    • Security configuration (process UID and GID, file access jail)

    • Process monitoring

    • Temporary file and directory management

    It is still possible to run both CloverDX and Apache Tomcat 9 packages but you need to take care of these requirements manually. For convenience, following script may be used to run either cloverdx or tomcat9 packages.

    Following script may be used as base for running CloverDX package but only reflects most basic configuration required for running the package and may require changes when default configuration of file locations is changed.

    #!/bin/sh
    
    APP=cloverdx
    
    set -a
    . /etc/$APP/clover.env
    set +a
    
    export TOMCAT_USER=clover
    export TOMCAT_GROUP=clover
    export CATALINA_PID=/run/$APP/pid
    
    start() {
      local cache="/var/cache/$APP"
      local log="/var/log/$APP"
      local temp="$CATALINA_TMPDIR"
      local rundir="`dirname $CATALINA_PID`"
    
      if id -u $TOMCAT_USER &>/dev/null; then
        # Create temp directory if not exists
        [ ! -d "$temp" ] && \
          mkdir -p "$temp" && \
          chown $TOMCAT_USER:$TOMCAT_GROUP "$temp"
    
        # Create cache directory if not exists
        [ ! -d "$cache" ] && \
          mkdir -p "$cache" && \
          chown $TOMCAT_USER:$TOMCAT_GROUP "$cache"
    
        # Create log directory if not exists
        [ ! -d "$log" ] && \
          mkdir -p "$log" && \
          chown $TOMCAT_USER:$TOMCAT_GROUP "$log"
    
        # Create run directory to be able to store PID file
        [ ! -d "$rundir" ] && \
          mkdir -p "$rundir" && \
          chown $TOMCAT_USER:$TOMCAT_GROUP "$rundir"
    
        /sbin/runuser -u $TOMCAT_USER $CATALINA_HOME/bin/startup.sh
        return $?
      else
        echo "User $TOMCAT_USER does not exist, exitting" >&2
        return 1
      fi
    }
    
    stop() {
      /sbin/runuser -u $TOMCAT_USER $CATALINA_HOME/bin/shutdown.sh
      return $?
    }
    
    case $1 in
    	start)
    	  start
    	;;
    	stop)
    	  stop
    	;;
    	restart)
    	  stop && start
    	;;
    	*)
    	  echo -e "Usage $0 start|stop|restart"
          exit 99
    	;;
    esac
    exit $?

    More sophisticated and robust solutions may be found tailored to specific process managers or implementing additional features like forced termination (e.g. https://gist.github.com/miglen/5590986). These solutions, in some cases, can be used without modification but testing in your specific environment is still strongly encouraged.