Version

    AWS Marketplace

    Overview

    The CloverDX Server offering on AWS Marketplace provides an easy way to create a CloverDX Server instance in the AWS cloud infrastructure. The offering spins-up a recommended cloud architecture that contains a standalone CloverDX Server with good defaults and recommended environment. The server uses PostgreSQL in RDS as its system database, which is automatically created and configured. The server instance uses AWS resources of the user, who is charged for them by Amazon.

    The CloverDX Server AWS offering consists of an AMI image of a virtual machine and of a CloudFormation template that provides a simple configuration user interface and creates the required cloud infrastructure as a CloudFormation stack.

    The following templates are provided:

    • CloverDX Server Deployment New VPC - deploys a new instance of CloverDX Server with newly created network infrastructure (VPC, Subnets etc.), see Quickstart. This is the easiest deployment option.

    • CloverDX Server Deployment Existing VPC - deploys a new instance of CloverDX Server into existing network infrastructure selected by the user, see Deployment Into Existing Infrastructure. This is the option to use if you already have existing infrastructure in AWS and want to integrate CloverDX with it.

    • CloverDX Server Upgrade - upgrades a CloverDX Server instance to a new version of CloverDX Server, see Upgrade in AWS.

    Quickstart

    Prerequisites:

    • license for

      CloverDX Server - this is a BYOL (Bring Your Own License) offering so you need to get your license from us - start here .

    • basic familiarity with AWS EC2 infrastructure

    • basic familiarity with AWS RDS infrastructure

    • basic familiarity with AWS CloudFormation

    • account on AWS and permissions to use EC2, RDS and CloudFormation services

    • key pair registered in EC2

    High-level overview of steps:

    1. Subscribe to CloverDX Server on AWS Marketplace

    2. Configure and deploy the CloudFormation stack

    3. Activate and configure the server instance

    Steps

    1. Subscribe to CloverDX Data Management Platform - Server BYOL offering on the AWS Marketplace, use the CloverDX Server Deployment New VPC Cloudformation template. Use the Continue to Subscribe button on the offering marketplace page and proceed through the wizard to the Launch action (meanwhile accepting Terms and Conditions, selecting CloverDX version and template etc).

    2. Deploy the CloudFormation stack

      1. First step: Specify template - a CloudFormation template is already selected from the marketplace offering, continue with Next

      2. Second step: Specify stack details - configure the stack:

        marketplace aws stack details
        Figure 14. Stack details

        • Stack name - unique name of the stack.

        • EC2 Instance type - type of EC2 instance where the server will run. A default instance type is pre-selected. You can select one of the supported instance types - larger instances have better performance but at a higher cost.

        • EC2 Availability zones - you must select 2 availability zones where the whole stack will be run. The first availability zone will be used for the CloverDX Server EC2 instance, both will be used for the system RDS database. An availability zone is basically a data center. Some availability zones might not support the selected EC2 instance type or the instance type might be unavailable due to capacity reasons - in such case select a different availability zone. Such issue would manifest during startup of the stack in its Events log.

        • Key Pair name - select a public key registered in your EC2 infrastructure. This key will be used to administer the EC2 instance via SSH.

        • Collect logs in CloudWatch - set to yes to enable centralized collection of logs to CloudWatch, see Server logs in AWS CloudWatch for more details.

        • Allow connections from - the CloudFormation template creates a Security Group automatically that allows only connections from specific IP ranges to specific ports on the instance. We recommend that you provide a range of IP addresses from which the instance should be available - typically your offices or data centers. For evaluation purposes you can use your public IP, obtained e.g. from myip.com. We do not recommend making the instance visible to the whole internet.

        • Admin user name - specify the user name of CloverDX Server administration user (or keep the default clover). This user is the first user available in Server Console, for administration of the server itself. This is NOT the operating system user - you must use the above public key to SSH to the instance.

        • Admin user password - specify the password for the above admin user.

        • Confirm admin user password - re-type the above password to confirm it.

        • Database master user name - specify the master user name of the RDS database that will be created as the server’s system database. This user name and below password will be used by CloverDX Server to connect to the database.

        • Database master user password - specify the password for the above database master user.

        • Confirm database master user password - re-type the above password to confirm it.

      3. Configure stack options and review the stack - next 2 pages allow you to set-up additional more advanced stack options, and review the whole stack configuration.

      4. Create stack - click the final Create Stack button to start the stack. Creation of the whole stack takes some time (up to a few minutes). You will see a CloudFormation log of resources being created. The stack is created and ready to use when it gets into the CREATE_COMPLETE state.

        marketplace aws stack deploy
        Figure 15. Stack deployment

    Success. CloverDX Server is now available in AWS. You can find its URL in the Outputs tab of the CloudFormation stack - the ServerURL entry. There you can also find its hostname for SSH access.

    marketplace aws stack outputs
    Figure 16. Stack outputs

    On the Server’s URL you will see the login page where you can:

    • Activate the server - the Server is licensed in BYOL (Bring Your Own License) mode, so you need to get your license from us - start here.

    • To login, use the credentials set in the CloudFormation configuration wizard.

      marketplace login page
      Figure 17. CloverDX Server login page

    The Server is running with default settings, and is immediately usable. It can be configured further to get it into full production quality.

    Architecture

    The CloverDX Server AWS offering consists of an AMI image of a virtual machine and of a CloudFormation template that orchestrates the required cloud resources:

    marketplace aws architecture
    Figure 18. Architecture - CloverDX Server in AWS marketplace

    Details of the AWS topology:

    • The instance runs in the AWS cloud, in the Region selected by user and in its 2 selected Availability Zones. The first AZ is used by the server’s EC2 instance, both AZs are used by the RDS database (for potential Multi-AZ deployment, see database details below).

    • A new VPC (Virtual Private Cloud) is created to isolate the CloverDX instance from other resources present in the AWS cloud. The VPC has an Internet Gateway for connectivity to the internet (also for the users to connect to the Server Console).

    • A new public Subnet is created in the VPC for the server’s network resources.

    • A single EC2 instance is created from an AMI. The

      CloverDX Server is running in this instance and uses an RDS instance of PostgreSQL, see below.

    • The EC2 instance is available via an Elastic IP - this IP address doesn’t change between restarts of the instance.

    • The EC2 instance uses a Security Group to limit access only to specific ports (22, 80, 443) and only from IP addresses from a defined CIDR range.

    • 2 new private Subnets are created in the VPC for the RDS database. The database currently uses just one of them, the second one is prepared for multi-AZ deployment of the database.

    • An RDS PosgreSQL database is created with recommended settings, and is used as the server’s system database.

    • The RDS database uses a Security Group to limit access only from the server’s EC2 instance.

    EC2 instance details (for additional information, see Common cloud architecture):

    • Operating system: Amazon Linux 2

    • AdoptOpenJDK 11

    • Tomcat 9

    • 2 disks - OS disk (10GB), data disk (128GB), both are Provisioned IOPS SSD EBS volumes. It is possible to increase the size of the disks or change the provisioned IOPS, just follow the documentation Encryption is enabled on both disks, see Security for more details.

    RDS database details:

    • Database type - PostgreSQL 11

    • DB instance class - db.m5.large (2 vCPUs, 8GB RAM)

    • Storage - 100GB SSD allocated storage, can autoscale to 200GB

    • Automated backups enabled - performed every day, backups retained for 7 days

    • Available only from the server’s EC2 instance, not visible from the Internet

    • Server uses SSL to communicate with the database

    • Multi-AZ deployment is disabled by default. Multi-AZ deployment runs multiple replicas of the database in different availability zones for high availability, which is why 2 availability zones have to be selected when creating the stack. The architecture and configuration is prepared for Multi-AZ deployment, and its possible to enable it in the RDS configuration later.

    Configuration

    For details about CloverDX Server configuration, see Common cloud configuration.

    Instance Type

    The EC2 instance type can be changed after the stack is deployed. To use a different instance type, stop the VM instance in EC2 console, change its instance type (via right-clicking > Instance settings > Change instance type) and start it again. It’s also possible to modify the CloudFormation template to use different instance types than the ones we offer by default. However, behavior and performance on other instance types than the default ones is not guaranteed.

    Memory

    Heap sizes for Server Core and Worker are set automatically based on the instance memory size, see Common cloud memory configuration. It is possible to change the instance type of the VM and the memory sizes will be re-calculated - for this stop the VM instance in EC2 console, change its instance type and start it again.

    Server logs

    By default, server logs are collected in /var/clover/cloverlogs directory, and Tomcat logs in /var/clover/tomcatlogs.

    The VM is pre-configured to send some specific logs to Amazon CloudWatch for centralized log collection and analysis. However, it is necessary to start the VM with an IAM role that allows it to send the logs to CloudWatch. The following steps create the IAM role and fully enable the CloudWatch integration:

    1. set the Collect logs in CloudWatch stack parameter to yes. The template will automatically create an IAM role that allows the EC2 instance to send the logs to CloudWatch.

      marketplace aws logs stack
      Figure 19. Collect logs in CloudWatch stack parameter

      This operation requires that the user starting the stack has permissions to manipulate with IAM roles. Example of IAM policy that allows the user to do that:

      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:CreateInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:PassRole",
                "iam:DetachRolePolicy",
                "iam:RemoveRoleFromInstanceProfile",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:AttachRolePolicy",
                "iam:PutRolePolicy",
                "iam:AddRoleToInstanceProfile"
            ],
            "Resource": "*"
        }
    ]
}

    2. acknowledge that the template might create IAM resources in the last step of launching the stack. If you don’t aknowledge that, you’ll see an error message Requires capabilities : [CAPABILITY_IAM].

      marketplace aws logs iam
      Figure 20. Confirmation that stack can create IAM resources

    The CloudWatch integration creates a log group called /cloverdx/stack-name, with the following logs:

    marketplace aws logs cloudwatch
    Figure 21. Logs in CloudWatch

    • all.log - main CloverDX Server log

    • performance.log - contains performance metrics that are useful for diagnostics of incidents, see Performance Log

    • job-queue.log - contains metrics useful for analysing behavior of the job queue, see Job Queue Log

    • worker.log - log of CloveDX Worker which executes jobs

    • catalina.out & catalina.log - Tomcat logs, useful primarily in case that CloverDX Server doesn’t start

    The CloudWatch integration is done via the CloudWatch Agent which is automatically configured and installed on the VM. Useful files and commands:

    • /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json - configuration of CloudWatch Agent, e.g. which logs are collected. Note that this file is on the OS disk, so changes to it will be overwritten when upgrading to new version (see Upgrade in AWS).

    • /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log - log file of the CloudWatch Agent, useful for diagnostics in casethe logs are not collected.

    • sudo systemctl restart amazon-cloudwatch-agent.service - restart the CloudWatch Agent service. This is needed after manual assignment of roleto the EC2 server instance or after changes to the above configuration file.

    The IAM role that allows CloudWatch integration can be assigned to the VM also manually in the EC2 console later. The role must allow write access to CloudWatch, e.g. by using the CloudWatchAgentServerPolicy managed policy. After assigning the role, you need to restart the CloudWatch Agent (see above).

    JMX monitoring

    To successfully establish a JMX connection over SSL to the server (e.g. for troubleshooting via VisualVM, see Common JMX Monitoring), the client needs to trust the server’s certificate. One option would be to replace the default server’s self-signed certificate with production-quality certificate (see HTTPS in AWS). In such case, no further configuration is required.

    To use JMX with the default self-signed certificate, follow these steps:

    1. Export the server’s certificate from the keystore:

      keytool -export -keystore $CLOVER_HOME/conf/serverCertificate.jks -storepass changeit -alias clover -rfc -file clover.cert

    2. Transfer the clover.cert certificate to the client machine.

    3. On the client machine, import the server certificate into client’s truststore:

      1. Into default Java truststore:

        sudo keytool -import -file clover.cert -alias clover -storetype JKS -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit

      2. Into a custom truststore:

        keytool -import -file clover.cert -alias clover -storetype JKS -keystore clover.truststore -storepass changeit

        In this case, the JMX client needs to be configured to use this custom truststore via properites:

        -Djavax.net.ssl.trustStore=clover.truststore -Djavax.net.ssl.trustStorePassword=changeit

    Users

    Users available in the EC2 instance:

    • ec2-user - standard AWS administrator user of the instance. Use sudo torun commands that require root privileges. Login via SSH using the public key selected when starting the stack.

    • root - not recommended to be used, cannot login to it directly.

    • clover - user that runs the CloverDX Server (i.e. it runs Tomcat). All files that CloverDX uses are owned by the clover user. It is not possible to login as clover

    Timezone

    The EC2 instance uses UTC timezone by default. The instance runs on Amazon Linux, so to change the timezone follow the documentation.

    Security

    This section describes security related aspects of the CloverDX Marketplace offering.

    Disk Encryption

    Both OS and data disks are encrypted by default via EBS encryption. No configuration is necessary, the disks are encrypted with the default key managed by AWS.

    Security Group

    The CloudFormation template creates a security group that serves as a virtual firewall. The security group allows connections to the following ports:

    • 22 - SSH

    • 80 - HTTP for Server Console and Server API

    • 443 - HTTPS for Server Console and Server API

    The connections are allowed only from the IP range specified when configuring the stack during its startup.

    The security group settings can be modified - you can find the security group in the Resources section of the CloudFormation stack and click on its configuration.

    HTTPS

    The CloverDX Server running in the stack has both HTTP and HTTPS enabled by default. You can find the server’s HTTPS URL in the Outputs section of the CloudFormation stack under ServerHttpsURL key. The HTTPS connector is running on port 443.

    The default HTTPS connector is using a self-signed certificate. So it is useful for encryption of communication between client (e.g. Designer) and the Server, but not for server identity verification. When connecting from a browser to the Server Console you will first see a warning about the certificate (e.g. Your connection is not private, NET::ERR_CERT_AUTHORITY_INVALID), after accepting this certificate you can work as usual and following accesses will not see this error. When first connecting from Designer you will need to accept the certificate.

    You can use your own production-quality certificate for the HTTPS connector:

    • Add your certificate to /var/clover/conf/serverCertificate.jks keystore (via the keytoolcommand), with an alias that is different from clover

    • Modify the /opt/clover/tomcat/conf/server.xmlTomcat configuration file to use the new certificate alias for its HTTPS connector.

    To disable usage of plain HTTP connectivity, modify the security group of the stack to block all connections to port 80.

    Deployment Into Existing Infrastructure

    If you’re already operating services and instances in AWS and want to deploy CloverDX Server alongside them, e.g. to integrate several data sources, then the deployment is performed with the CloudFormation template CloverDX Server Deployment Existing VPC. The template does not create a new VPC, Subnets and some other networking infrastructure, but lets you select existing ones. The template creates a new CloverDX Server EC2 instance and a new RDS system database and deploys them into the selected network resources.

    Deployment steps and settings in this case are the same as described in Quickstart with these differences:

    1. Subscribe to CloverDX Data Management Platform - Server BYOL offering on the AWS Marketplace - use the CloverDX Server Deployment Existing VPC template

    2. Configure the stack:

      marketplace aws existing infra stack details
      Figure 22. Deployment to existing infrastructure stack details

      • VPC - select a VPC where the stack will be deployed.

      • Subnet - select a Subnet which will be used by the CloverDX Server EC2 instance. The Subnet must be in the VPC selected above.

      • Database Subnets - select 2 Subnets which will be used by the RDS system database. Typically the subnets are private and not visible from the internet. The subnets must be in the VPC selected above.

      • Security Group - select a Security Group which will be used by the CloverDX Server EC2 instance. The RDS system database will use its own Security Group that is created by the template and allows connections only from the Server.

    Upgrade

    To upgrade CloverDX Server to a new version in AWS Marketplace, use the CloverDX Server Upgrade template. The upgrade template creates a new stack with the new version of CloverDX and possibly updated cloud architecture. Configuration and data are re-used from the previous version by using snapshots of data disk and RDS system database. The previous version has only minimal downtime and can run in parallel with the new version.

    First see an overview of the upgrade process in Common cloud upgrade. This section assumes familiarity with deploying a new CloverDX Server as described in Quickstart. The upgrade template is also similar to the template for deployment into existing infrastructure (see Deployment Into Existing Infrastructure).

    The upgrade process is supported when upgrading from CloverDX version 5.6.1 or newer.

    High-level overview::

    1. Create snapshots of the data disk and RDS database of the previous version

    2. Subscribe to new version of CloverDX Server on AWS Marketplace using the CloverDX Server Upgrade template

    3. Configure and deploy the CloudFormation stack of the new version, using the above snapshots

    4. Test the new version

    Upgrade steps

    The below steps focus on the new or different actions done during the upgrade process and they assume knowledge of the deployment of a new CloverDX Server stack (see Quickstart).

    1. Pause processing of jobs in the previous version - disable listeners and schedules, this starts downtime of the previous version.

    2. Create a snapshot of the system RDS database of the previous version - you can find the RDS database in the Resources section of your stack. After clicking on it, use the Take Snapshot action on the database.

      marketplace aws upgrade rds select
      Figure 23. Selecting RDS system database in the stack
      marketplace aws upgrade rds snapshot
      Figure 24. Creating snapshot of the RDS system database

    3. Create a snapshot of the data disk of the previous version - in the Resources section of your stack, you can find the EC2 instance. After clicking on it, in the EC2 details you can click on the /dev/sdf volume to take you to the EBS volume to create the snapshot.

      marketplace aws upgrade ec2 select
      Figure 25. Selecting CloverDX Server EC2 instance
      marketplace aws upgrade ebs select
      Figure 26. Selecting data disk EBS volume

    4. Resume processing of jobs in the previous version - this ends the downtime of the previous version.

    5. Subscribe to the new version of CloverDX Data Management Platform - Server BYOL offering on the AWS Marketplace, use the CloverDX Server Upgrade Cloudformation template.

    6. Configure the stack of the new version:

      marketplace aws upgrade stack details
      Figure 27. Upgrade stack details

      • VPC - select the VPC where the previous version is running. If you used the CloverDX Server Deployment New VPC template to deploy the previous version, then you can find the VPC by its name “stackname-VPC” (“stackname” is the name of the stack of the previous version).

      • Subnet - select the Subnet where the previous version EC2 instance with CloverDX Server is running. You can find the Subnet by its name “stackname-Subnet” if you deployed the previous version via the CloverDX Server Deployment New VPC template.

      • Database Subnets - select the 2 Subnets where the previous version RDS system database is running. You can find the Subnets by their names “stackname-DBPrivateSubnet1” and “stackname-DBPrivateSubnet2” if you deployed the previous version via the CloverDX Server Deployment New VPC template.

      • Security Group - select the Security Group used by the previous version’s EC2 instance. You can find the Security Group by its name “stackname-SecurityGroup” if you deployed the previous version via the CloverDX Server Deployment New VPC template.

      • Data disk snapshot - ID of the snapshot of the data disk of the previous version that you created in one of the previous steps. You can find it in the EC2 console in the Snapshots section after selecting the snapshot.

        marketplace aws upgrade datadisk snapshot id
        Figure 28. Data disk snapshot ID

      • RDS database snapshot - ARN of the snapshot of the RDS system database of the previous version that you created in one of the previous steps. You can find it in the RDS console in the Snapshots section after selecting the snapshot.

        marketplace aws upgrade rds snapshot arn
        Figure 29. RDS system database snapshot ARN

      • Note that you’re not configuring the credentials of the CloverDX Server admin user and database credentials - they are the same as in the previous version.

    Success. New version of CloverDX Server is now available in AWS. You can find its URL in the Outputs tab of the CloudFormation stack. Login credentials for the new version are the same as for the previous version. The new version is deployed in a new version of the cloud architecture, while re-using the data disk and RDS system database of the previous version based on their snapshots. The previous version is left untouched and can be continued to be used.

    Follow-up steps

    • Activate the new server - you will probably need to provide a license for the new CloverDX Server version. You can do this on the login page.

    • Test the new version - perform thorough testing of the new version. If you disabled the listeners and schedules on the previous version, then the new version starts up and does not process new data. You need to carefully enable processing that can be enabled, e.g. if it doesn’t consume production data.

    • After finishing testing of the new version it is possible to switch the Elastic IP address of the previous version to point to the new version instance. This allows you to continue using the same IP address or hostname with the new version, e.g. for Server projects in CloverDX Designer.

      Steps to switch Elastic IP address:

      1. Disassociate the new Elastic IP - un-bind the new Elastic IP from the new version instance. Select the Elastic IP of the new version (e.g. from the Resources tab of the stack of the new version) and use the Disassociate Elastic IP address action. Warning: this will cause the new version instance to be available on a different IP address than previously (one will be assigned automatically by AWS).

      2. Release the new Elastic IP - the above disassociated Elastic IP is currently unused and you’re being charged for it. So release it via the Release Elastic IP address action.

      3. Associate the old Elastic IP to the new version instance - to redirect the old Elastic IP address to the new version instance, select it (e.g. from the Resources tab of the stack of the previous version) and use the Associate Elastic IP address. Warning: this will cause the previous version instance to be available on a different IP address than previously (one will be assigned automatically by AWS).

      4. For SSL connectivity to the new version to work correctly, we need to re-generate the self-signed certificate to correspond to the new public hostname. For this just restart the new version instance.