CloverDX Designer
CloverDX Overview
Family of CloverDX Products
Products Introduction
CloverDX Designer
CloverDX Server
CloverDX Cluster
Online Resources
Installation
System Requirements
Installation
Related Links
Downloading
Installing
Upgrading
Starting
Activating
License Manager
CloverDX License Dialog
IBM InfoSphere MDM Plugin Installation
Downloading
Requirements
Installation into Designer
Troubleshooting
Installation into Server
Troubleshooting
Troubleshooting
Windows
Windows SmartScreen
User Account Control
Windows 10 Firewall
Linux
Designer Stops Responding
Welcome Page not Displayed
Hints on Edges Have no Content
Component Editor Not Working
Others
Subclipse
Online Activity
Configuration
Configuration
Runtime Configuration
Logging
Master Password
User Classpath
CloverDX Server Integration
Ignored Files
Execution Monitoring
Java Configuration
Engine Configuration
Refresh Operation
Using Designer
Designer User Interface
Designer Panes
Graph Editor with Palette of Components
Navigator Pane
Outline Pane
Tabs Pane
Execution Tab
Keyboard Shortcuts
Projects
Types of CloverDX Projects
CloverDX (Local) Project
CloverDX Server Project
Creating CloverDX Projects
CloverDX Project
CloverDX Server Project
CloverDX Examples Project
Converting CloverDX Projects
Converting Local Project to Server Project
Converting Server Projects to Local Project
Structure of CloverDX Projects
Standard Structure of All CloverDX Projects
The .classpath File
Workspace.prm File
Versioning of Server Project Content
Initial Check-Out of Project from Repository
Adding Server Project to Version Control
Connecting Server Project to Existing Repository
Getting Changes from Repository
Committing into Repository
Working with CloverDX Server Projects
Working Offline
Handling Conflicts
Project Configuration
CloverDX Connection
Ignored Files
Graphs
Creating an Empty Graph
Creating a Simple Graph
Execution
Successful Graph Execution
Run Configuration
Main Tab
Parameters Tab
Refresh Tab
Connecting to a Running Job
Graph States
Common Dialogs
URL File Dialog
Local Files
Workspace View
CloverDX Server
Hadoop HDFS
Remote Files
Port
Dictionary
Filtering Files and Tips
Edit Value Dialog
Open Type Dialog
Import
Import CloverDX Projects
Import CloverDX Project from URL
Import CloverDX Projects
Import from CloverDX Server Sandbox
Import Graphs
Import Metadata
Metadata from XSD
Metadata from DDL
Export
Convert Graph to Jobflow
Convert Jobflow to Graph
Convert Subgraph to Graph
Export Graphs to HTML
Export to CloverDX Server Sandbox
Export Image
Graph Tracking
Changing Record Count Font Size
Search Functionality
Working with CloverDX Server
CloverDX Server Project Basic Principles
Connecting via HTTP
Connecting via HTTPS
Designer has its Own Certificate
Designer does not have its Own Certificate
Connecting via Proxy Server
Graphs
Components
Adding Components
Finding Components
Edit Component Dialog
Commands
Attributes
Enable/Disable Component
Enabling Component
Disabling Component
Enabling by Graph Parameter
Enabling by Connected Input Port
Disable as Trash
Passing Data Through Disabled Component
Common Properties of Components
Component Name
Phases
Component Allocation
Specific Attribute Types
Time Intervals
Group Key
Sort Key
Metadata Templates
Edges
Connecting Components with Edges
Types of Edges
Colors of Edges
Debugging Edges
Selecting Debug Data
Viewing Debug Data
Turning Off Debug
Edge Memory Allocation
Metadata
Records and Fields
Record Types
Data Types in Metadata
Data Formats
Locale and Locale Sensitivity
Time Zone
Autofilling Functions
Metadata Types
Internal Metadata
External (Shared) Metadata
SQL Query Metadata
Reading Metadata from Special Sources
Auto-propagated Metadata
Priorities of Metadata
Propagation of SQL Query Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Extracting Metadata from Salesforce
SQL Query Metadata
User Defined Metadata
Merging Existing Metadata
Creating Database Table from Metadata and Database Connection
Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Changing Record Delimiter
Changing Default (Field) Delimiter
Defining Non-Default Delimiter for a Field
Editing Metadata in the Source Code
Multivalue Fields
Lists and Maps Support in Components
Joining on Lists and Maps (Comparison Rules)
Connections
Database Connections
Internal Database Connections
External (Shared) Database Connections
Database Connection Properties
Encryption of Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Hive Connection
Troubleshooting
JMS Connections
Internal JMS Connections
External (Shared) JMS Connections
Edit JMS Connection Wizard
Encrypting the Authentication Password
QuickBase Connections
Hadoop Connections
Connecting to YARN
Libraries Needed for Hadoop
Kerberos Authentication for Hadoop
Kafka Connections
Creating Kafka Connection
Details
MongoDB Connections
Salesforce Connections
Creating Salesforce Connection
Important Details
Lookup Tables
Lookup Tables in Cluster Environment
Internal Lookup Tables
Creating Internal Lookup Tables
Externalizing Internal Lookup Tables
Exporting Internal Lookup Tables
External (Shared) Lookup Tables
Creating External (Shared) Lookup Tables
Linking External (Shared) Lookup Tables
Internalizing External (Shared) Lookup Tables
Types of Lookup Tables
Simple Lookup Table
Database Lookup Table
Range Lookup Table
Persistent Lookup Table
Aspell Lookup Table
Sequences
Persistent Sequences
Non Persistent Sequences
Internal Sequences
Creating Internal Sequences
Externalizing Internal Sequences
Exporting Internal Sequences
External (Shared) Sequences
Creating External (Shared) Sequences
Linking External (Shared) Sequences
Internalizing External (Shared) Sequences
Editing a Sequence
Sequences in Cluster Environment
Parameters
Internal Parameters
Creating Internal Parameters
Externalizing Internal Parameters
External (Shared) Parameters
Creating External Parameters
Linking External Parameters
Internalizing External (Shared) Parameters
XML Schema of External Parameters
Graph Parameter Editor
Secure Graph Parameters
Parameters with CTL2 Expressions (Dynamic Parameters)
Environment Variables
Canonicalizing File Paths
Using Parameters
Internal/External Graph Elements
Internal Graph Elements
External (Shared) Graph Elements
Working with Graph Elements
Advantages of External (Shared) Graph Elements
Advantages of Internal Graph Elements
Changing Form of Graph Elements
Dictionary
Creating a Dictionary
Dictionary Entry Types
Dictionary Entry Properties
Using a Dictionary in Graphs
Accessing Dictionary from Readers and Writers
Accessing Dictionary with Java
Accessing Dictionary with CTL2
Execution Properties
Notes in Graphs
Placing Notes into Graph
Resizing Notes
Editing Notes
Formatted Text
Links from Notes
Folding Notes
Notes Properties
Compatibility
Transformations
Defining Transformations
Components Allowing Transformation
Java or CTL
Internal or External Definition
Return Values of Transformations
Error Actions and Error Log (deprecated since 3.0)
Transform Editor
Common Java Interfaces
Data Partitioning (Parallel Running)
Data Partitioning in Cluster
High Availability
Scalability
Transformation Requests
Parallel Data Processing
Graph Allocation Examples
Example of Distributed Execution
Details of the Example Transformation Design
Scalability of the Example Transformation
Remote Edges
Subgraphs
Subgraphs Overview
Subgraphs Introduction
Design & Execution
Subgraphs vs. Jobflow
Using Subgraphs
Using Subgraphs
Configuring Subgraphs
Developing Subgraphs
Wrapping
Creating from Scratch
Making Subgraph Configurable
Developing and Testing Subgraphs
Filling Required Parameters
Metadata Propagation
Design Patterns
Readers
Writers
Transformers
Executors
Jobflow
Jobflow Overview
Jobflow Introduction
What is CloverDX Jobflow?
Design and Execution
Anatomy of the Jobflow Module
Important Concepts
Dynamic Attribute Setting
Parameter Passing
Pass-Through Mapping
Execution Status Reporting
Error Handling
Jobflow Execution Model: Single Token
Jobflow Execution Model: Multiple Tokens
Stopping on Error
Synchronous vs. Asynchronous Execution
Logging
Advanced Concepts
Daemon Jobs
Killing Jobs
Jobflow Design Patterns
Data Services
Data Services Introduction
Architecture
Development
Data Service Job Editor
Endpoint Configuration
Data Service REST Job Logic
Anatomy of Data Service Jobs
Input and Output Components
HTTP Request Payload
HTTP Request Parameters
HTTP Headers
HTTP Response
Multiple Edges
HTTP Status Code and Headers
Execution Steps of Data Service Jobs
Exceptions and Error Handling
Auto-generated Documentation and Swagger/OpenAPI Definition
Testing
Testing Service Logic in Designer
Testing Services Deployed on Server
Use cases
Custom Serialization
Using Static File as Response
Sending a File Generated by .rjob
Publishing a Static File
Using CTL2 Functions in Data Services
Data Service that Receives a File or Text in Body Part
Converting Graph to Data Service
Converting Graphs to Data Service
Publishing Data Service
Publishing Multiple Data Services at Once
Unpublishing Data Service
Unpublishing Multiple Data Services at Once
Example
Echo to Upper Case
Troubleshooting
Server Returns Error Code 404
Server Returns Error Code 500
Server Returns Error Code 503
Component Reference
Readers
Common Properties of Readers
Supported File URL Formats for Readers
Viewing Data on Readers
Input Port Reading
Incremental Reading
Selecting Input Records
Data Policy
XML Features
CTL Templates for Readers
Java Interfaces for Readers
CloverDataReader
ComplexDataReader
CustomJavaReader
DatabaseReader
DataGenerator
DBFDataReader
EmailReader
FlatFileReader
HadoopReader
JavaBeanReader
JMSReader
JSONExtract
JSONReader
LDAPReader
MongoDBReader
MultiLevelReader
ParallelReader
QuickBaseRecordReader
QuickBaseQueryReader
SalesforceBulkReader
SalesforceReader
SpreadsheetDataReader
UniversalDataReader
XMLExtract
XMLReader
XMLXPathReader
Writers
Common Properties of Writers
Supported File URL Formats for Writers
Viewing Data on Writers
Output Port Writing
Appending or Overwriting
Creating Directories
Selecting Output Records
Partitioning Output into Different Output Files
Excluding Fields
Java Interfaces for Writers
CloverDataWriter
CustomJavaWriter
DatabaseWriter
DB2BulkWriter
DBFDataWriter
EmailSender
FlatFileWriter
HadoopWriter
InfobrightBulkWriter
InformixBulkWriter
JavaBeanWriter
JavaMapWriter
JMSWriter
JSONWriter
LDAPWriter
MongoDBWriter
MSSQLBulkWriter
MySQLBulkWriter
OracleBulkWriter
PostgreSQLBulkWriter
QuickBaseImportCSV
QuickBaseRecordWriter
SalesforceBulkWriter
SalesforceWriter
SalesforceWaveWriter
SpreadsheetDataWriter
StructuredDataWriter
TableauWriter
Trash
UniversalDataWriter
XMLWriter
Transformers
Common Properties of Transformers
CTL Templates for Transformers
Java Interfaces for Transformers
Aggregate
Concatenate
CustomJavaTransformer
DataIntersection
DataSampler
Dedup
Denormalizer
ExtSort
FastSort
Filter
LoadBalancingPartition
Merge
MetaPivot
Normalizer
Partition
Pivot
Reformat
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
Common Properties of Joiners
Join Types
Slave Duplicates
CTL Templates for Joiners
Java Interfaces for Joiners
Combine
CrossJoin
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Job Control
Common Properties of Job Control
Barrier
Condition
ExecuteGraph
ExecuteJobflow
ExecuteMapReduce
ExecuteProfilerJob
ExecuteScript
Fail
GetJobInput
KillGraph
KillJobflow
Loop
MonitorGraph
MonitorJobflow
SetJobOutput
Sleep
Subgraph
Success
TokenGather
File Operations
Common Properties of File Operations
Common Attributes of File Operation Components
Supported URL Formats for File Operations
CopyFiles
CreateFiles
DeleteFiles
ListFiles
MoveFiles
Data Partitioning
Common Properties of Data Partitioning Components
ParallelLoadBalancingPartition
ParallelMerge
ParallelPartition
ParallelRepartition
ParallelSimpleCopy
ParallelSimpleGather
Data Quality
Common Properties of Data Quality
EmailFilter
ProfilerProbe
Validator
List of Rules
See also
Others
Common Properties of Others
CheckForeignKey
CustomJavaComponent
DBExecute
HTTPConnector
LookupTableReaderWriter
RunGraph
SequenceChecker
SystemExecute
WebServiceClient
Deprecated
ApproximativeJoin
JavaExecute
MongoDBExecute
Incubation
KafkaCommit
KafkaReader
KafkaWriter
CTL2 - CloverDX Transformation Language
CTL Overview
Basic Features of CTL
CTL History
Language Reference
Program Structure
Comments
Import
Metadata Import in CTL2
Data Types in CTL2
byte
cbyte
date
decimal
integer
long
number (double)
string
list
map
variant
record
Literals
Variables
Constants
Dictionary in CTL2
Operators
Arithmetic Operators
Relational Operators
Logical Operators
Assignment Operator
Ternary Operator
Conditional Fail Expression
Simple Statement and Block of Statements
Control Statements
Conditional Statements
Iteration Statements
Jump Statements
Error Handling
Try-Catch Statement
OnError() Functions
Functions
Message Function
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Regular Expressions
CTL Debugging
Debug Perspective
Importing and Exporting Breakpoints
Exporting Breakpoints
Importing Breakpoints
Inspecting Variables and Expressions
Inspect Action
Expressions View and Watch Action
Examples
Basic Example
Using Hit Count
Conditional Breakpoint
Detecting Changes of the Value
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Mapping Functions
Container Functions
Record Functions (Dynamic Field Access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Subgraph Functions
Data Service HTTP Library Functions
Custom CTL Functions
List of All CTL2 Functions
CTL2 Appendix - List of National-specific Characters
Tutorial
Creating a Transformation Graph
Terminology
Creating a Project
Creating a New Data File
Creating a Graph
Placing Components in the Graph Editor Pane
Connecting Components by an Edge
Extracting Metadata from the Input File
Assigning Metadata to the Edges
Setting Up Readers (FlatFileReader)
Setting Up Writers (SpreadsheetDataWriter)
Running the Graph
Opening the Output File
What to do next
Filtering the records
Inserting the Filter
Setting Up the Filter Component
Two data streams
Sorting the Records
Adding ExtSort
Setting Up the ExtSort Component
Processing Speedup with Parallelization
Debugging the Java Transformation
Debugging the Java Transformation in Local Projects
Debugging the Java Transformation in Server Projects
CloverDX Server
Introduction
What is CloverDX Server?
CloverDX Server Architecture
CloverDX Core
CloverDX Worker
CloverDX Cluster
Installation
System Requirements
Installing
Evaluation Server
Installation
Production Server
Apache Tomcat
IBM WebSphere
JBoss Enterprise Application Platform
Oracle WebLogic Server
Activation
CloverDX Server Activation Alternatives
IBM InfoSphere MDM Plugin Installation
Installation into Server
Postinstallation Configuration
Memory Settings
Maximum Number of Open Files
Maximum Number of Processes or Threads
Firewall Exceptions
Garbage Collector for Worker
Reverse Proxy Configuration
Optional Installation Steps
Adding Libraries to the Worker's Classpath
Troubleshooting
Upgrade
Cloud Marketplaces
AWS Marketplace
Overview
Quickstart
Architecture
Configuration
Security
Deployment Into Existing Infrastructure
Upgrade
Azure Marketplace
Overview
Quickstart
Architecture
Configuration
Security
Deployment Into Existing Infrastructure
Upgrade
Configuration
Configuration Introduction
Configuration Sources
Configuration File on Specified Location
Alternative Configuration Sources
Priorities of Configuration Sources
Specifying the Path to the Configuration File
Setup
Before You Start
Using Setup
Configuration File
License
Database
Worker
Sandboxes
Encryption
E-Mail
LDAP
Cluster
System Database Configuration
Embedded Apache Derby
MySQL
Creating Database
CloverDX Server Setup
DB2
Creating Database
CloverDX Server Setup
DB2 on AS/400
Oracle
Creating Database
CloverDX Server Setup
Microsoft SQL Server
Creating Database
CloverDX Server Setup
PostgreSQL
Creating Database
CloverDX Server Setup
JNDI Configuration and Encryption
JNDI DB Datasource
JNDI Datasource Troubleshooting
Encrypted JNDI
List of Configuration Properties
General Configuration Properties
Worker - Configuration Properties
Worker Health Related Properties
Worker - JNDI Properties
JDBC Datasources
JMS Connections
Worker - SSL Properties
Properties on Worker's Command Line
Job Execution Properties
List of all properties
Secure Configuration Properties
Logging
Main Logs
Graph Run Logs
Server Audit Logs
Designer-Server Integration Logs
Performance Log
Job Queue Log
Monitor Log
Administration
Operations Dashboard
Overview
Quickstart
Using the Dashboard
Logs
Scenarios
API
Configuration
Limitations
Monitoring
Resources
Cluster Overview
Node Detail
Server Logs
Using the Monitoring
Restarting the Worker
Showing Worker's Command Line Arguments
Suspending the Server
Resuming the Server
Displaying List of Threads of the Server Core
Dismissing Warning Messages
Temp Space Management
Temp Space Overview
Management
Secure Parameters
Users and Groups
LDAP Authentication
Active Directory
SAML Authentication
SAML configuration examples
SAML troubleshooting
Users
Groups
User Lockout
Sandboxes - Server Side Job Files
Referencing Files from the Graph or Jobflow
Sandbox Content Security and Permissions
Sandbox Content and Options
Download sandbox as ZIP
Upload ZIP to sandbox
Download file in ZIP
Create File
Create Directory
Upload File
Export Sandbox Configuration
Delete Sandbox
Sandbox Configuration Import
Execution Properties
WebDAV Access to Sandboxes
WebDAV Clients
WebDAV Authentication/Authorization
Server Configuration Migration
Server Configuration Export
Server Configuration Import
Environment Variables
Passwords
Upgrading Server to Newer Version
Support Package
Diagnostics
Troubleshooting Worker
Alerts and Notification
Using Graphs
Job Queue
Overview
Quickstart
Architecture
Load Metrics
Emergency Mode
Job Queue Algorithm
Impact
Scenarios
Troubleshooting
Configuration
Limitations
Graph/Jobflow Parameters
Parameters by Execution Type
Adding Another Graph Parameters
Tasks
Send an Email
Placeholders
Execute Shell Command
Start a Graph
Start a Jobflow
Start a Profiler Job
Abort job
Archive Records
Send a JMS Message
Execute Groovy Code
Manual Task Execution
Scheduling
Timetable Setting
Allocations of Scheduled Task on Nodes
Scheduling the Tasks - Examples
Start a graph at specific time
Start a Jobflow once an hour
Complex Scheduling
Alerts and Notification
Viewing Job Runs - Execution History
Filtering and ordering
Tracking
Log File
Listeners
Graph Event Listeners
Graph Events
Listener
Tasks
Use Cases
Jobflow Event Listeners
Jobflow Events
Listener
Tasks
JMS Message Listeners
Universal Event Listeners
Evaluation Criteria
File Event Listeners (remote and local)
Cluster environment
Supported filesystems and protocols
Observed file
File Events
Check Interval, Task and Use Cases
Howtos
Task Failure Listeners
Task Choice
Task Failed E-mail Template
Alerts and Notification
Recommendations for Transformations Developers
Extensibility - CloverDX Engine Plugins
Troubleshooting
API
Simple HTTP API
REST API
Overview
Data Services
Data Services Overview
User Interface
Endpoints
List of Data Services
Overview
Data App
State and History
Alerts and Notification
Configuration
Catalog of Services
Built-in Data Service Examples
HTTPS Connectors
Using Data Services
Deploying Data Service
Publishing and Unpublishing Data Service from Sandbox
Publishing Data Service Examples
Changing Data Service to Anonymous
Running Data Service on HTTPS
Running Data Service on HTTPS on Cluster
Monitoring Data Service
Testing Data Service
Performance Tuning
Exporting Data Service Configuration
Importing Data Service Configuration
Avoiding Premature Marking of Data Service as Failing
Looking up Particular Data Service
Resetting State of Failing Data Service Endpoint
Enabling CORS Filter
Custom HTTP Headers
Data Services on Cluster
Data Apps
Create new Data App
Using Data Apps
Validation
Customizing Data Apps
Examples, How to, Best practices
Branding of Data Apps
SOAP WebService API
SOAP WS Client
SOAP WS API Authentication/Authorization
JMX mBean
JMX Configuration
How to configure JMX on Apache Tomcat
How to Configure JMX on WebSphere
Possible Problems
Operations
Launch Services
Launch Services Authentication
Sending the Data to Launch Service
Results of the Graph Execution
Cluster
Sandboxes in Cluster
Using a Sandbox Resource as a Component Data Source
Remote Edges
Cluster Configuration
Mandatory Cluster Properties
Optional Cluster Properties
Example of 2 Node Cluster Configuration
Basic 2-nodes Cluster Configuration
2-nodes Cluster with Proxied Access to Database
2-nodes Cluster with Load Balancer
Example of 3 Node Cluster Configuration
Basic 3-nodes Cluster Configuration
Jobs Load Balancing Properties
Running More Clusters
Recommendations for Cluster Deployment
Troubleshooting
NodeA Cannot Establish HTTP Connection to NodeB
NodeA Cannot Establish TCP Connection (Port 7800 by Default) to NodeB
NodeB is Killed or It Cannot Connect to the Database
Auto-Resuming in Unreliable Network
Long-Term Network Malfunction May Cause Jobs to Hang on
Security
Security Recommendations for CloverDX Server
CloverDX Profiler
About CloverDX Data Profiler
Installation
Hardware Requirements
Software Requirements
Licensing
Getting CloverDX Data Profiler
Configuration
Run Configurations
Log level
Preferences
CloverDX Data Profiler
Results Database
Using Profiler
CloverDX Data Profiler GUI
CloverDX Data Profiler essentials
Working With Projects
Profiling Session Step-by-step
Profiling Jobs
Creating Jobs
File Profiling
Database Table Profiling
Excel Sheet Profiling
Metrics
Data Sampling
Modifying Jobs
Running Jobs
Reporting Console
Screens hierarchy
Job List
Job View
Run View
Field View
Reporting Console as Standalone Server
Integration with CloverDX Server
Hardware and Software Requirements
Installation
Tomcat
WebSphere
WebLogic
JBoss
Default Configuration
Configuration
Base Configuration
Alternative ways of configuration
Database Support
Derby
MSSQL
MySQL
Oracle
PostgreSQL
JNDI
Configuring CloverDX Server
Connection pooling
User Accounts Settings
Permissions
Group-level permissions
Public API Authorization
Public API
Historic Metric Values
RSS Feed
Profiler and CloverDX Graphs
Data Profiler Graph Components
ProfilerProbe
ExecuteProfilerJob
Profiler-Designer Element Sharing
Sharing From Designer to Profiler
Sharing From Profiler to Designer
List of Metrics
Version
CloverDX Designer
>
Configuration
Part III. Configuration
Table of Contents
12. Configuration
13. Runtime Configuration
Logging
Master Password
User Classpath
14. CloverDX Server Integration
Ignored Files
15. Execution Monitoring
16. Java Configuration
17. Engine Configuration
18. Refresh Operation
