CloverDX Designer

Table of Contents

I. Overview
1. Family of CloverDX Products
Introduction
CloverDX Designer
CloverDX Server
CloverDX Cluster
2. Online Resources
II. Installation
3. System Requirements
Installation
Related Links
4. Downloading
5. Installing
6. Upgrading
7. Starting
8. Activating
Activation Using License Key
Activation Online
9. License Manager
CloverDX License Dialog
10. IBM InfoSphere MDM Plugin Installation
Downloading
Requirements
Installation into Designer
Troubleshooting
Installation into Server
Troubleshooting
11. Optional Installation Steps
Support for SMB 2.x and 3.x
CloverDX Designer
CloverDX Server
12. Troubleshooting
Windows
Windows SmartScreen
User Account Control
Windows 10 Firewall
Linux
Designer Stops Responding
Welcome Page not Displayed
Hints on Edges Have no Content
Others
Subclipse
III. Configuration
13. Configuration
14. Runtime Configuration
Logging
Master Password
User Classpath
15. CloverDX Server Integration
Ignored Files
16. Execution Monitoring
17. Java Configuration
18. Engine Configuration
19. Refresh Operation
IV. Using Designer
20. Designer User Interface
Designer Panes
Graph Editor with Palette of Components
Navigator Pane
Outline Pane
Tabs Pane
Execution Tab
Keyboard Shortcuts
21. Projects
Types of CloverDX Projects
CloverDX (Local) Project
CloverDX Server Project
Creating CloverDX Projects
CloverDX Project
CloverDX Server Project
CloverDX Examples Project
Converting CloverDX Projects
Converting Local Project to Server Project
Converting Server Projects to Local Project
Structure of CloverDX Projects
Standard Structure of All CloverDX Projects
The .classpath File
Workspace.prm File
Versioning of Server Project Content
Initial Check-Out of Project from Repository
Adding Server Project to Version Control
Connecting Server Project to Existing Repository
Getting Changes from Repository
Committing into Repository
Working with CloverDX Server Projects
Working Offline
Handling Conflicts
Project Configuration
CloverDX Connection
Ignored Files
22. Graphs
Creating an Empty Graph
Creating a Simple Graph
23. Execution
Successful Graph Execution
Run Configuration
Main Tab
Parameters Tab
Refresh Tab
Connecting to a Running Job
Graph States
24. Common Dialogs
URL File Dialog
Local Files
Workspace View
CloverDX Server
Hadoop HDFS
Remote Files
Port
Dictionary
Filtering Files and Tips
Edit Value Dialog
Open Type Dialog
25. Import
Import CloverDX Projects
Import from CloverDX Server Sandbox
Import Graphs
Import Metadata
Metadata from XSD
Metadata from DDL
26. Export
Convert Graph to Jobflow
Convert Jobflow to Graph
Convert Subgraph to Graph
Export Graphs to HTML
Export to CloverDX Server Sandbox
Export Image
27. Graph Tracking
Changing Record Count Font Size
28. Search Functionality
29. Working with CloverDX Server
CloverDX Server Project Basic Principles
Connecting via HTTP
Connecting via HTTPS
Designer has its Own Certificate
Designer does not have its Own Certificate
Connecting via Proxy Server
V. Graphs
30. Components
Adding Components
Finding Components
Edit Component Dialog
Commands
Attributes
Enable/Disable Component
Enabling Component
Disabling Component
Enabling by Graph Parameter
Enabling by Connected Input Port
Disable as Trash
Passing Data Through Disabled Component
Common Properties of Components
Component Name
Phases
Component Allocation
Specific Attribute Types
Time Intervals
Group Key
Sort Key
Metadata Templates
31. Edges
Connecting Components with Edges
Types of Edges
Assigning Metadata to Edges
Colors of Edges
Debugging Edges
Selecting Debug Data
Viewing Debug Data
Turning Off Debug
Edge Memory Allocation
32. Metadata
Records and Fields
Record Types
Data Types in Metadata
Data Formats
Locale and Locale Sensitivity
Time Zone
Autofilling Functions
Metadata Types
Internal Metadata
External (Shared) Metadata
Dynamic Metadata
Reading Metadata from Special Sources
Auto-propagated Metadata
Priorities of Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Extracting Metadata from Salesforce
Extracting Metadata from Lotus Notes
User Defined Metadata
Merging Existing Metadata
Creating Database Table from Metadata and Database Connection
Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Changing Record Delimiter
Changing Default (Field) Delimiter
Defining Non-Default Delimiter for a Field
Editing Metadata in the Source Code
Multivalue Fields
Lists and Maps Support in Components
Joining on Lists and Maps (Comparison Rules)
33. Connections
Database Connections
Internal Database Connections
External (Shared) Database Connections
Database Connection Properties
Encryption of Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Hive Connection
Troubleshooting
JMS Connections
Internal JMS Connections
External (Shared) JMS Connections
Edit JMS Connection Wizard
Encrypting the Authentication Password
QuickBase Connections
Lotus Connections
Hadoop connection
Libraries Needed for Hadoop
Kerberos Authentication for Hadoop
MongoDB connection
Salesforce connection
Creating Salesforce Connection
Important Details
34. Lookup Tables
LookupTables in Cluster Environment
Internal Lookup Tables
Creating Internal Lookup Tables
Externalizing Internal Lookup Tables
Exporting Internal Lookup Tables
External (Shared) Lookup Tables
Creating External (Shared) Lookup Tables
Linking External (Shared) Lookup Tables
Internalizing External (Shared) Lookup Tables
Types of Lookup Tables
Simple Lookup Table
Database Lookup Table
Range Lookup Table
Persistent Lookup Table
Aspell Lookup Table
35. Sequences
Persistent Sequences
Non Persistent Sequences
Internal Sequences
Creating Internal Sequences
Externalizing Internal Sequences
Exporting Internal Sequences
External (Shared) Sequences
Creating External (Shared) Sequences
Linking External (Shared) Sequences
Internalizing External (Shared) Sequences
Editing a Sequence
Sequences in Cluster Environment
36. Parameters
Internal Parameters
Externalizing Internal Parameters
External (Shared) Parameters
Creating External Parameters
Linking External Parameters
Internalizing External (Shared) Parameters
XML Schema of External Parameters
Graph Parameter Editor
Secure Graph Parameters
Parameters with CTL Expressions (Dynamic Parameters)
Environment Variables
Canonicalizing File Paths
Using Parameters
37. Internal/External Graph Elements
Internal Graph Elements
External (Shared) Graph Elements
Working with Graph Elements
Advantages of External (Shared) Graph Elements
Advantages of Internal Graph Elements
Changing Form of Graph Elements
38. Dictionary
Creating a Dictionary
Using a Dictionary in Graphs
Accessing Dictionary from Readers and Writers
Accessing Dictionary with Java
Accessing Dictionary with CTL2
39. Notes in Graphs
Placing Notes into Graph
Resizing Notes
Editing Notes
Formatted Text
Links from Notes
Folding Notes
Notes Properties
Compatibility
40. Transformations
Defining Transformations
Components Allowing Transformation
Java or CTL
Internal or External Definition
Return Values of Transformations
Error Actions and Error Log (deprecated since 3.0)
Transform Editor
Common Java Interfaces
41. Data Partitioning (Parallel Running)
42. Data Partitioning in Cluster
High Availability
Scalability
Transformation Requests
Parallel Data Processing
Graph Allocation Examples
Example of Distributed Execution
Details of the Example Transformation Design
Scalability of the Example Transformation
Remote Edges
VI. Subgraphs
43. Overview
Introduction
Design & Execution
Subgraphs vs. Jobflow
44. Using Subgraphs
Using Subgraphs
Configuring Subgraphs
45. Developing Subgraphs
Wrapping
Creating from Scratch
Making Subgraph Configurable
Developing and Testing Subgraphs
Filling Required Parameters
Metadata Propagation
46. Design Patterns
Readers
Writers
Transformers
Executors
VII. Jobflow
47. Jobflow Overview
Introduction
What is CloverDX Jobflow?
Design and Execution
Anatomy of the Jobflow Module
Important Concepts
Dynamic Attribute Setting
Parameter Passing
Pass-Through Mapping
Execution Status Reporting
Error Handling
Jobflow Execution Model: Single Token
Jobflow Execution Model: Multiple Tokens
Stopping on Error
Synchronous vs. Asynchronous Execution
Logging
Advanced Concepts
Daemon Jobs
Killing Jobs
48. Jobflow Design Patterns
VIII. Data Services
49. Overview
50. Architecture
51. Development
Data Service Job Editor
Endpoint Configuration
Data Service REST Job Logic
Anatomy of Data Service Jobs
Input and Output Components
HTTP Request Payload
HTTP Request Parameters
HTTP Headers
HTTP Response
Multiple Edges
HTTP Status Code and Headers
Execution Steps of Data Service Jobs
Exceptions and Error Handling
Auto-generated Documentation and Swagger/OpenAPI Definition
Testing
Testing Service Logic in Designer
Testing Services Deployed on Server
52. Use cases
Custom Serialization
Sending a File Generated by .rjob
Publishing a Static File
Using CTL2 Functions in Data Services
Data Service that Receives a File or Text in Body Part
Converting Graph to Data Service
Converting Graphs to Data Service
Publishing Data Service
Publishing Multiple Data Services at Once
Unpublishing Data Service
Unpublishing Multiple Data Services at Once
53. Example
Echo to Upper Case
54. Troubleshooting
Server Returns Error Code 404
Server Returns Error Code 500
Server Returns Error Code 503
IX. Component Reference
55. Readers
Common Properties of Readers
Supported File URL Formats for Readers
Viewing Data on Readers
Input Port Reading
Incremental Reading
Selecting Input Records
Data Policy
XML Features
CTL Templates for Readers
Java Interfaces for Readers
CloverDataReader
ComplexDataReader
CustomJavaReader
DataGenerator
DBFDataReader
DBInputTable
EmailReader
FlatFileReader
HadoopReader
JavaBeanReader
JMSReader
JSONExtract
JSONReader
LDAPReader
LotusReader
MongoDBReader
MultiLevelReader
ParallelReader
QuickBaseRecordReader
QuickBaseQueryReader
SalesforceBulkReader
SalesforceReader
SpreadsheetDataReader
UniversalDataReader
XMLExtract
XMLReader
XMLXPathReader
56. Writers
Common Properties of Writers
Supported File URL Formats for Writers
Viewing Data on Writers
Output Port Writing
Appending or Overwriting
Creating Directories
Selecting Output Records
Partitioning Output into Different Output Files
Excluding Fields
Java Interfaces for Writers
CloverDataWriter
CustomJavaWriter
DB2DataWriter
DBFDataWriter
DBOutputTable
EmailSender
FlatFileWriter
HadoopWriter
InfobrightDataWriter
InformixDataWriter
JavaBeanWriter
JavaMapWriter
JMSWriter
JSONWriter
LDAPWriter
LotusWriter
MongoDBWriter
MSSQLDataWriter
MySQLDataWriter
OracleDataWriter
PostgreSQLDataWriter
QuickBaseImportCSV
QuickBaseRecordWriter
SalesforceBulkWriter
SalesforceWriter
SalesforceWaveWriter
SpreadsheetDataWriter
StructuredDataWriter
TableauWriter
Trash
UniversalDataWriter
XMLWriter
57. Transformers
Common Properties of Transformers
CTL Templates for Transformers
Java Interfaces for Transformers
Aggregate
Concatenate
CustomJavaTransformer
DataIntersection
DataSampler
Dedup
Denormalizer
ExtSort
FastSort
Filter
LoadBalancingPartition
Merge
MetaPivot
Normalizer
Partition
Pivot
Reformat
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
58. Joiners
Common Properties of Joiners
Join Types
Slave Duplicates
CTL Templates for Joiners
Java Interfaces for Joiners
Combine
CrossJoin
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
59. Job Control
Common Properties of Job Control
Barrier
Condition
ExecuteGraph
ExecuteJobflow
ExecuteMapReduce
ExecuteProfilerJob
ExecuteScript
Fail
GetJobInput
KillGraph
KillJobflow
Loop
MonitorGraph
MonitorJobflow
SetJobOutput
Sleep
Subgraph
Success
TokenGather
60. File Operations
Common Properties of File Operations
Supported URL Formats for File Operations
CopyFiles
CreateFiles
DeleteFiles
ListFiles
MoveFiles
61. Data Partitioning
Common Properties of Data Partitioning Components
ParallelLoadBalancingPartition
ParallelMerge
ParallelPartition
ParallelRepartition
ParallelSimpleCopy
ParallelSimpleGather
62. Data Quality
Common Properties of Data Quality
AddressDoctor 5
EmailFilter
ProfilerProbe
Validator
List of Rules
63. Others
Common Properties of Others
CheckForeignKey
CustomJavaComponent
DBExecute
HTTPConnector
LookupTableReaderWriter
MongoDBExecute
RunGraph
SequenceChecker
SystemExecute
WebServiceClient
64. Deprecated
ApproximativeJoin
JavaExecute
X. CTL2 - CloverDX Transformation Language
65. Overview
66. Language Reference
Program Structure
Comments
Import
Data Types in CTL2
byte
cbyte
date
decimal
integer
long
number (double)
string
list
map
record
Literals
Variables
Dictionary in CTL2
Operators
Arithmetic Operators
Relational Operators
Logical Operators
Assignment Operator
Ternary Operator
Conditional Fail Expression
Simple Statement and Block of Statements
Control Statements
Conditional Statements
Iteration Statements
Jump Statements
Error Handling
Functions
Message Function
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Regular Expressions
67. CTL Debugging
Debug Perspective
Importing and Exporting Breakpoints
Exporting Breakpoints
Importing Breakpoints
Inspecting Variables and Expressions
Inspect Action
Expressions View and Watch Action
Examples
Basic Example
Using Hit Count
Conditional Breakpoint
Detecting Changes of the Value
68. Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Mapping Functions
Container Functions
Record Functions (Dynamic Field Access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Subgraph Functions
Data Service HTTP Library Functions
Custom CTL Functions
List of All CTL2 Functions
CTL2 Appendix - List of National-specific Characters
List of Figures
List of Tables
List of Examples