Version

    CloverDX Designer

    User's Guide

    This User's Guide refers to CloverDX Designer 5.5.x release.

    Copyright © 2010-2020 CloverDX a.s. All rights reserved.

    Do not copy or distribute without express permission of CloverDX a.s.

    Feedback welcome:

    If you have any comments or suggestions for this documentation, please send them by email to .

    Consider How to speed up communication with CloverCARE support before contacting the support.

    Table of Contents

    I. CloverDX Overview
    1. Family of CloverDX Products
    Products Introduction
    CloverDX Designer
    CloverDX Server
    CloverDX Cluster
    2. Online Resources
    II. Installation
    3. System Requirements
    Installation
    Related Links
    4. Downloading
    5. Installing
    6. Upgrading
    7. Starting
    8. Activating
    9. License Manager
    CloverDX License Dialog
    10. IBM InfoSphere MDM Plugin Installation
    Downloading
    Requirements
    Installation into Designer
    Troubleshooting
    Installation into Server
    Troubleshooting
    11. Troubleshooting
    Windows
    Windows SmartScreen
    User Account Control
    Windows 10 Firewall
    Linux
    Designer Stops Responding
    Welcome Page not Displayed
    Hints on Edges Have no Content
    Component Editor Not Working
    Others
    Subclipse
    III. Configuration
    12. Configuration
    13. Runtime Configuration
    Logging
    Master Password
    User Classpath
    14. CloverDX Server Integration
    Ignored Files
    15. Execution Monitoring
    16. Java Configuration
    17. Engine Configuration
    18. Refresh Operation
    IV. Using Designer
    19. Designer User Interface
    Designer Panes
    Graph Editor with Palette of Components
    Navigator Pane
    Outline Pane
    Tabs Pane
    Execution Tab
    Keyboard Shortcuts
    20. Projects
    Types of CloverDX Projects
    CloverDX (Local) Project
    CloverDX Server Project
    Creating CloverDX Projects
    CloverDX Project
    CloverDX Server Project
    CloverDX Examples Project
    Converting CloverDX Projects
    Converting Local Project to Server Project
    Converting Server Projects to Local Project
    Structure of CloverDX Projects
    Standard Structure of All CloverDX Projects
    The .classpath File
    Workspace.prm File
    Versioning of Server Project Content
    Initial Check-Out of Project from Repository
    Adding Server Project to Version Control
    Connecting Server Project to Existing Repository
    Getting Changes from Repository
    Committing into Repository
    Working with CloverDX Server Projects
    Working Offline
    Handling Conflicts
    Project Configuration
    CloverDX Connection
    Ignored Files
    21. Graphs
    Creating an Empty Graph
    Creating a Simple Graph
    22. Execution
    Successful Graph Execution
    Run Configuration
    Main Tab
    Parameters Tab
    Refresh Tab
    Connecting to a Running Job
    Graph States
    23. Common Dialogs
    URL File Dialog
    Local Files
    Workspace View
    CloverDX Server
    Hadoop HDFS
    Remote Files
    Port
    Dictionary
    Filtering Files and Tips
    Edit Value Dialog
    Open Type Dialog
    24. Import
    Import CloverDX Projects
    Import CloverDX Project from URL
    Import CloverDX Projects
    Import from CloverDX Server Sandbox
    Import Graphs
    Import Metadata
    Metadata from XSD
    Metadata from DDL
    25. Export
    Convert Graph to Jobflow
    Convert Jobflow to Graph
    Convert Subgraph to Graph
    Export Graphs to HTML
    Export to CloverDX Server Sandbox
    Export Image
    26. Graph Tracking
    Changing Record Count Font Size
    27. Search Functionality
    28. Working with CloverDX Server
    CloverDX Server Project Basic Principles
    Connecting via HTTP
    Connecting via HTTPS
    Designer has its Own Certificate
    Designer does not have its Own Certificate
    Connecting via Proxy Server
    V. Graphs
    29. Components
    Adding Components
    Finding Components
    Edit Component Dialog
    Commands
    Attributes
    Enable/Disable Component
    Enabling Component
    Disabling Component
    Enabling by Graph Parameter
    Enabling by Connected Input Port
    Disable as Trash
    Passing Data Through Disabled Component
    Common Properties of Components
    Component Name
    Phases
    Component Allocation
    Specific Attribute Types
    Time Intervals
    Group Key
    Sort Key
    Metadata Templates
    30. Edges
    Connecting Components with Edges
    Types of Edges
    Assigning Metadata to Edges
    Colors of Edges
    Debugging Edges
    Selecting Debug Data
    Viewing Debug Data
    Turning Off Debug
    Edge Memory Allocation
    31. Metadata
    Records and Fields
    Record Types
    Data Types in Metadata
    Data Formats
    Locale and Locale Sensitivity
    Time Zone
    Autofilling Functions
    Metadata Types
    Internal Metadata
    External (Shared) Metadata
    SQL Query Metadata
    Reading Metadata from Special Sources
    Auto-propagated Metadata
    Priorities of Metadata
    Propagation of SQL Query Metadata
    Creating Metadata
    Extracting Metadata from a Flat File
    Extracting Metadata from an XLS(X) File
    Extracting Metadata from a Database
    Extracting Metadata from a DBase File
    Extracting Metadata from Salesforce
    SQL Query Metadata
    User Defined Metadata
    Merging Existing Metadata
    Creating Database Table from Metadata and Database Connection
    Metadata Editor
    Basics of Metadata Editor
    Record Pane
    Field Name vs. Label vs. Description
    Details Pane
    Changing and Defining Delimiters
    Changing Record Delimiter
    Changing Default (Field) Delimiter
    Defining Non-Default Delimiter for a Field
    Editing Metadata in the Source Code
    Multivalue Fields
    Lists and Maps Support in Components
    Joining on Lists and Maps (Comparison Rules)
    32. Connections
    Database Connections
    Internal Database Connections
    External (Shared) Database Connections
    Database Connection Properties
    Encryption of Access Password
    Browsing Database and Extracting Metadata from Database Tables
    Windows Authentication on Microsoft SQL Server
    Hive Connection
    Troubleshooting
    JMS Connections
    Internal JMS Connections
    External (Shared) JMS Connections
    Edit JMS Connection Wizard
    Encrypting the Authentication Password
    QuickBase Connections
    Hadoop connection
    Connecting to YARN
    Libraries Needed for Hadoop
    Kerberos Authentication for Hadoop
    MongoDB connection
    Salesforce connection
    Creating Salesforce Connection
    Important Details
    33. Lookup Tables
    Lookup Tables in Cluster Environment
    Internal Lookup Tables
    Creating Internal Lookup Tables
    Externalizing Internal Lookup Tables
    Exporting Internal Lookup Tables
    External (Shared) Lookup Tables
    Creating External (Shared) Lookup Tables
    Linking External (Shared) Lookup Tables
    Internalizing External (Shared) Lookup Tables
    Types of Lookup Tables
    Simple Lookup Table
    Database Lookup Table
    Range Lookup Table
    Persistent Lookup Table
    Aspell Lookup Table
    34. Sequences
    Persistent Sequences
    Non Persistent Sequences
    Internal Sequences
    Creating Internal Sequences
    Externalizing Internal Sequences
    Exporting Internal Sequences
    External (Shared) Sequences
    Creating External (Shared) Sequences
    Linking External (Shared) Sequences
    Internalizing External (Shared) Sequences
    Editing a Sequence
    Sequences in Cluster Environment
    35. Parameters
    Internal Parameters
    Externalizing Internal Parameters
    External (Shared) Parameters
    Creating External Parameters
    Linking External Parameters
    Internalizing External (Shared) Parameters
    XML Schema of External Parameters
    Graph Parameter Editor
    Secure Graph Parameters
    Parameters with CTL2 Expressions (Dynamic Parameters)
    Environment Variables
    Canonicalizing File Paths
    Using Parameters
    36. Internal/External Graph Elements
    Internal Graph Elements
    External (Shared) Graph Elements
    Working with Graph Elements
    Advantages of External (Shared) Graph Elements
    Advantages of Internal Graph Elements
    Changing Form of Graph Elements
    37. Dictionary
    Creating a Dictionary
    Using a Dictionary in Graphs
    Accessing Dictionary from Readers and Writers
    Accessing Dictionary with Java
    Accessing Dictionary with CTL2
    38. Notes in Graphs
    Placing Notes into Graph
    Resizing Notes
    Editing Notes
    Formatted Text
    Links from Notes
    Folding Notes
    Notes Properties
    Compatibility
    39. Transformations
    Defining Transformations
    Components Allowing Transformation
    Java or CTL
    Internal or External Definition
    Return Values of Transformations
    Error Actions and Error Log (deprecated since 3.0)
    Transform Editor
    Common Java Interfaces
    40. Data Partitioning (Parallel Running)
    41. Data Partitioning in Cluster
    High Availability
    Scalability
    Transformation Requests
    Parallel Data Processing
    Graph Allocation Examples
    Example of Distributed Execution
    Details of the Example Transformation Design
    Scalability of the Example Transformation
    Remote Edges
    VI. Subgraphs
    42. Subgraphs Overview
    Subgraphs Introduction
    Design & Execution
    Subgraphs vs. Jobflow
    43. Using Subgraphs
    Using Subgraphs
    Configuring Subgraphs
    44. Developing Subgraphs
    Wrapping
    Creating from Scratch
    Making Subgraph Configurable
    Developing and Testing Subgraphs
    Filling Required Parameters
    Metadata Propagation
    45. Design Patterns
    Readers
    Writers
    Transformers
    Executors
    VII. Jobflow
    46. Jobflow Overview
    Jobflow Introduction
    What is CloverDX Jobflow?
    Design and Execution
    Anatomy of the Jobflow Module
    Important Concepts
    Dynamic Attribute Setting
    Parameter Passing
    Pass-Through Mapping
    Execution Status Reporting
    Error Handling
    Jobflow Execution Model: Single Token
    Jobflow Execution Model: Multiple Tokens
    Stopping on Error
    Synchronous vs. Asynchronous Execution
    Logging
    Advanced Concepts
    Daemon Jobs
    Killing Jobs
    47. Jobflow Design Patterns
    VIII. Data Services
    48. Data Services Introduction
    49. Architecture
    50. Development
    Data Service Job Editor
    Endpoint Configuration
    Data Service REST Job Logic
    Anatomy of Data Service Jobs
    Input and Output Components
    HTTP Request Payload
    HTTP Request Parameters
    HTTP Headers
    HTTP Response
    Multiple Edges
    HTTP Status Code and Headers
    Execution Steps of Data Service Jobs
    Exceptions and Error Handling
    Auto-generated Documentation and Swagger/OpenAPI Definition
    Testing
    Testing Service Logic in Designer
    Testing Services Deployed on Server
    51. Use cases
    Custom Serialization
    Using Static File as Response
    Sending a File Generated by .rjob
    Publishing a Static File
    Using CTL2 Functions in Data Services
    Data Service that Receives a File or Text in Body Part
    Converting Graph to Data Service
    Converting Graphs to Data Service
    Publishing Data Service
    Publishing Multiple Data Services at Once
    Unpublishing Data Service
    Unpublishing Multiple Data Services at Once
    52. Example
    Echo to Upper Case
    53. Troubleshooting
    Server Returns Error Code 404
    Server Returns Error Code 500
    Server Returns Error Code 503
    IX. Component Reference
    54. Readers
    Common Properties of Readers
    Supported File URL Formats for Readers
    Viewing Data on Readers
    Input Port Reading
    Incremental Reading
    Selecting Input Records
    Data Policy
    XML Features
    CTL Templates for Readers
    Java Interfaces for Readers
    CloverDataReader
    ComplexDataReader
    CustomJavaReader
    DatabaseReader
    DataGenerator
    DBFDataReader
    EmailReader
    FlatFileReader
    HadoopReader
    JavaBeanReader
    JMSReader
    JSONExtract
    JSONReader
    LDAPReader
    MongoDBReader
    MultiLevelReader
    ParallelReader
    QuickBaseRecordReader
    QuickBaseQueryReader
    SalesforceBulkReader
    SalesforceReader
    SpreadsheetDataReader
    UniversalDataReader
    XMLExtract
    XMLReader
    XMLXPathReader
    55. Writers
    Common Properties of Writers
    Supported File URL Formats for Writers
    Viewing Data on Writers
    Output Port Writing
    Appending or Overwriting
    Creating Directories
    Selecting Output Records
    Partitioning Output into Different Output Files
    Excluding Fields
    Java Interfaces for Writers
    CloverDataWriter
    CustomJavaWriter
    DatabaseWriter
    DB2BulkWriter
    DBFDataWriter
    EmailSender
    FlatFileWriter
    HadoopWriter
    InfobrightBulkWriter
    InformixBulkWriter
    JavaBeanWriter
    JavaMapWriter
    JMSWriter
    JSONWriter
    LDAPWriter
    MongoDBWriter
    MSSQLBulkWriter
    MySQLBulkWriter
    OracleBulkWriter
    PostgreSQLBulkWriter
    QuickBaseImportCSV
    QuickBaseRecordWriter
    SalesforceBulkWriter
    SalesforceWriter
    SalesforceWaveWriter
    SpreadsheetDataWriter
    StructuredDataWriter
    TableauWriter
    Trash
    UniversalDataWriter
    XMLWriter
    56. Transformers
    Common Properties of Transformers
    CTL Templates for Transformers
    Java Interfaces for Transformers
    Aggregate
    Concatenate
    CustomJavaTransformer
    DataIntersection
    DataSampler
    Dedup
    Denormalizer
    ExtSort
    FastSort
    Filter
    LoadBalancingPartition
    Merge
    MetaPivot
    Normalizer
    Partition
    Pivot
    Reformat
    Rollup
    SimpleCopy
    SimpleGather
    SortWithinGroups
    XSLTransformer
    57. Joiners
    Common Properties of Joiners
    Join Types
    Slave Duplicates
    CTL Templates for Joiners
    Java Interfaces for Joiners
    Combine
    CrossJoin
    DBJoin
    ExtHashJoin
    ExtMergeJoin
    LookupJoin
    RelationalJoin
    58. Job Control
    Common Properties of Job Control
    Barrier
    Condition
    ExecuteGraph
    ExecuteJobflow
    ExecuteMapReduce
    ExecuteProfilerJob
    ExecuteScript
    Fail
    GetJobInput
    KillGraph
    KillJobflow
    Loop
    MonitorGraph
    MonitorJobflow
    SetJobOutput
    Sleep
    Subgraph
    Success
    TokenGather
    59. File Operations
    Common Properties of File Operations
    Common Attributes of File Operation Components
    Supported URL Formats for File Operations
    CopyFiles
    CreateFiles
    DeleteFiles
    ListFiles
    MoveFiles
    60. Data Partitioning
    Common Properties of Data Partitioning Components
    ParallelLoadBalancingPartition
    ParallelMerge
    ParallelPartition
    ParallelRepartition
    ParallelSimpleCopy
    ParallelSimpleGather
    61. Data Quality
    Common Properties of Data Quality
    EmailFilter
    ProfilerProbe
    Validator
    List of Rules
    62. Others
    Common Properties of Others
    CheckForeignKey
    CustomJavaComponent
    DBExecute
    HTTPConnector
    LookupTableReaderWriter
    RunGraph
    SequenceChecker
    SystemExecute
    WebServiceClient
    63. Deprecated
    ApproximativeJoin
    JavaExecute
    MongoDBExecute
    X. CTL2 - CloverDX Transformation Language
    64. CTL Overview
    Basic Features of CTL
    CTL History
    65. Language Reference
    Program Structure
    Comments
    Import
    Data Types in CTL2
    byte
    cbyte
    date
    decimal
    integer
    long
    number (double)
    string
    list
    map
    record
    Literals
    Variables
    Dictionary in CTL2
    Operators
    Arithmetic Operators
    Relational Operators
    Logical Operators
    Assignment Operator
    Ternary Operator
    Conditional Fail Expression
    Simple Statement and Block of Statements
    Control Statements
    Conditional Statements
    Iteration Statements
    Jump Statements
    Error Handling
    Functions
    Message Function
    Conditional Fail Expression
    Accessing Data Records and Fields
    Mapping
    Parameters
    Regular Expressions
    66. CTL Debugging
    Debug Perspective
    Importing and Exporting Breakpoints
    Exporting Breakpoints
    Importing Breakpoints
    Inspecting Variables and Expressions
    Inspect Action
    Expressions View and Watch Action
    Examples
    Basic Example
    Using Hit Count
    Conditional Breakpoint
    Detecting Changes of the Value
    67. Functions Reference
    Conversion Functions
    Date Functions
    Mathematical Functions
    String Functions
    Mapping Functions
    Container Functions
    Record Functions (Dynamic Field Access)
    Miscellaneous Functions
    Lookup Table Functions
    Sequence Functions
    Subgraph Functions
    Data Service HTTP Library Functions
    Custom CTL Functions
    List of All CTL2 Functions
    CTL2 Appendix - List of National-specific Characters