Version

    CloverDX Designer

    User’s Guide

    This User’s Guide refers to CloverDX Designer 6.5.0 release.

    Copyright © 2010-2024 CloverDX a.s. All rights reserved.

    Do not copy or distribute without express permission of CloverDX a.s.

    CloverDX Overview
    1. Family of CloverDX Products
    Products Introduction
    CloverDX Designer
    CloverDX Server
    CloverDX Cluster
    2. Online Resources
    Installation
    3. System Requirements
    4. Downloading
    5. Installing
    Windows
    Mac OS
    Linux
    6. Starting
    7. Activating
    8. License Manager
    CloverDX License Dialog
    9. Troubleshooting
    Windows
    Windows SmartScreen
    User Account Control
    Windows 10 Firewall
    Linux
    Designer Stops Responding
    Welcome Page not Displayed
    Hints on Edges Have no Content
    Component Editor Not Working
    Others
    Subclipse
    Online Activity
    Logs
    10. IBM InfoSphere MDM Plugin Installation
    Downloading
    Installation into Designer
    Troubleshooting
    Installation into Server
    Troubleshooting
    Upgrading
    Configuration
    11. Configuration
    12. Runtime Configuration
    Logging
    Master Password
    User Classpath
    13. CloverDX Server Integration
    Ignored Files
    14. Execution Monitoring
    15. Java Configuration
    16. Engine Configuration
    17. Refresh Operation
    Using Designer
    18. Designer User Interface
    Designer Panes
    Graph Editor with Palette of Components
    Project Explorer Pane
    Outline Pane
    Tabs Pane
    Execution Tab
    Keyboard Shortcuts
    19. Projects
    Types of CloverDX Projects
    CloverDX (Local) Project
    CloverDX Server Project
    Creating CloverDX Projects
    CloverDX Project
    CloverDX Server Project
    CloverDX Examples Project
    Converting CloverDX Projects
    Converting Local Project to Server Project
    Converting Server Projects to Local Project
    Structure of CloverDX Projects
    Standard Structure of All CloverDX Projects
    The .classpath File
    Workspace.prm File
    Versioning of Server Project Content
    Initial Check-Out of Project from Repository
    Adding Server Project to Version Control
    Connecting Server Project to Existing Repository
    Getting Changes from Repository
    Committing into Repository
    Working with CloverDX Server Projects
    Working Offline
    Handling Conflicts
    Project Configuration
    CloverDX Connection
    Ignored Files
    20. Graphs
    Creating an Empty Graph
    Creating a Simple Graph
    21. Execution
    Successful Graph Execution
    Run Configuration
    Run Configuration vs CloverDX Runtime
    Main Tab
    Parameters Tab
    Refresh Tab
    Connecting to a Running Job
    Graph States
    22. Common Dialogs
    URL File Dialog
    Local Files
    Workspace View
    Hadoop HDFS
    Remote Files
    Port
    Dictionary
    Filtering Files and Tips
    Edit Value Dialog
    Open Type Dialog
    23. Import
    Import CloverDX Projects
    Import CloverDX Project from URL
    Import CloverDX Projects
    Import from CloverDX Server Sandbox
    Import Graphs
    Import Metadata
    Metadata from XSD
    Metadata from DDL
    Import Wrangler Mapping
    Reject metadata
    24. Export
    Convert Graph to Jobflow
    Convert Jobflow to Graph
    Convert Subgraph to Graph
    Export Graphs to HTML
    Export to CloverDX Server Sandbox
    Export Image
    25. Export Project as Library
    26. Graph Tracking
    Changing Record Count Font Size
    27. Search Functionality
    28. Working with CloverDX Server
    CloverDX Server Project Basic Principles
    Connecting via HTTP
    Connecting via HTTPS
    Designer has its Own Certificate
    Designer does not have its Own Certificate
    Connecting via Proxy Server
    Graphs
    29. Components
    Adding Components
    Finding Components
    Edit Component Dialog
    Commands
    Attributes
    Enable/Disable Component
    Enabling Component
    Disabling Component
    Enabling by Graph Parameter
    Enabling by Connected Input Port
    Disable as Trash
    Passing Data Through Disabled Component
    Common Properties of Components
    Component Name
    Phases
    Component Allocation
    Specific Attribute Types
    Time Intervals
    Group Key
    Sort Key
    Metadata Templates
    30. Edges
    Connecting Components with Edges
    Types of Edges
    Assigning Metadata to Edges
    Colors of Edges
    Debugging Edges
    Selecting Debug Data
    Viewing Debug Data
    Turning Off Debug
    Edge Memory Allocation
    31. Metadata
    Records and Fields
    Record Types
    Data Types in Metadata
    Data Formats
    Locale and Locale Sensitivity
    Time Zone
    Autofilling Functions
    Metadata Types
    Internal Metadata
    External (Shared) Metadata
    SQL Query Metadata
    Reading Metadata from Special Sources
    Auto-propagated Metadata
    Priorities of Metadata
    Propagation of SQL Query Metadata
    Creating Metadata
    Extracting Metadata from a Flat File
    Extracting Metadata from an XLS(X) File
    Extracting Metadata from a Database
    Extracting Metadata from a DBase File
    Extracting Metadata from Salesforce
    SQL Query Metadata
    User Defined Metadata
    Merging Existing Metadata
    Creating Database Table from Metadata and Database Connection
    Metadata Editor
    Basics of Metadata Editor
    Record Pane
    Field Name vs. Label vs. Description
    Details Pane
    Changing and Defining Delimiters
    Changing Record Delimiter
    Changing Default (Field) Delimiter
    Defining Non-Default Delimiter for a Field
    Editing Metadata in the Source Code
    Multivalue Fields
    Lists and Maps Support in Components
    Joining on multivalue fields (Comparison Rules)
    32. Connections
    Database Connections
    Internal Database Connections
    External (Shared) Database Connections
    Database Connection Properties
    Encryption of Access Password
    Browsing Database and Extracting Metadata from Database Tables
    Windows Authentication on Microsoft SQL Server
    Snowflake Connection
    Hive Connection
    Troubleshooting
    JMS Connections
    Internal JMS Connections
    External (Shared) JMS Connections
    Edit JMS Connection Wizard
    Encrypting the Authentication Password
    QuickBase Connections
    Hadoop Connections
    Connecting to YARN
    Libraries Needed for Hadoop
    Kerberos Authentication for Hadoop
    Kafka Connections
    Creating Kafka Connection
    Details
    OAuth2 Connections
    Creating OAuth2 Connection
    Authorize connection
    Details
    MongoDB Connections
    Salesforce Connections
    Creating Salesforce Connection
    Important Details
    33. Lookup Tables
    Lookup Tables in Cluster Environment
    Internal Lookup Tables
    Creating Internal Lookup Tables
    Externalizing Internal Lookup Tables
    Exporting Internal Lookup Tables
    External (Shared) Lookup Tables
    Creating External (Shared) Lookup Tables
    Linking External (Shared) Lookup Tables
    Internalizing External (Shared) Lookup Tables
    Types of Lookup Tables
    Simple Lookup Table
    Database Lookup Table
    Range Lookup Table
    Persistent Lookup Table
    Aspell Lookup Table
    34. Sequences
    Persistent Sequences
    Non-Persistent Sequences
    Internal Sequences
    Creating Internal Sequences
    Externalizing Internal Sequences
    Exporting Internal Sequences
    External (Shared) Sequences
    Creating External (Shared) Sequences
    Linking External (Shared) Sequences
    Internalizing External (Shared) Sequences
    Editing a Sequence
    Sequences in Cluster Environment
    35. Parameters
    Internal Parameters
    Creating Internal Parameters
    Externalizing Internal Parameters
    External (Shared) Parameters
    Creating External Parameters
    Linking External Parameters
    Internalizing External (Shared) Parameters
    XML Schema of External Parameters
    Graph Parameter Editor
    Secure Graph Parameters
    Parameters with CTL2 Expressions (Dynamic Parameters)
    Environment Variables
    Canonicalizing File Paths
    Using Parameters
    36. Internal/External Graph Elements
    Internal Graph Elements
    External (Shared) Graph Elements
    Working with Graph Elements
    Advantages of External (Shared) Graph Elements
    Advantages of Internal Graph Elements
    Changing Form of Graph Elements
    37. Dictionary
    Creating a Dictionary
    Dictionary Entry Types
    Dictionary Entry Properties
    Using a Dictionary in Graphs
    Accessing Dictionary from Readers and Writers
    Accessing Dictionary with Java
    Accessing Dictionary with CTL2
    38. Execution Properties
    39. Notes in Graphs
    Placing Notes into Graph
    Resizing Notes
    Editing Notes
    Formatted Text
    Links from Notes
    Folding Notes
    Notes Properties
    Compatibility
    40. Transformations
    Defining Transformations
    Components Allowing Transformation
    Java or CTL
    Internal or External Definition
    Return Values of Transformations
    Error Actions and Error Log (deprecated since 3.0)
    Transform Editor
    Common Java Interfaces
    Debugging Java Transformations
    Debugging Java Transformations in Local Projects
    Debugging Java Transformation in Server Projects
    41. Data Partitioning (Parallel Running)
    42. Data Partitioning in Cluster
    High Availability
    Scalability
    Transformation Requests
    Parallel Data Processing
    Graph Allocation Examples
    Example of Distributed Execution
    Details of the Example Transformation Design
    Scalability of the Example Transformation
    Remote Edges
    Subgraphs
    43. Subgraphs Overview
    Subgraphs Introduction
    Design & Execution
    Subgraphs vs. Jobflow
    44. Using Subgraphs
    Using Subgraphs
    Configuring Subgraphs
    45. Developing Subgraphs
    Wrapping
    Creating from Scratch
    Making Subgraph Configurable
    Developing and Testing Subgraphs
    Filling Required Parameters
    Metadata Propagation
    46. Design Patterns
    Readers
    Writers
    Transformers
    Executors
    Jobflow
    47. Jobflow Overview
    Jobflow Introduction
    What is CloverDX Jobflow?
    Design and Execution
    Anatomy of the Jobflow Module
    Important Concepts
    Dynamic Attribute Setting
    Parameter Passing
    Pass-Through Mapping
    Execution Status Reporting
    Error Handling
    Jobflow Execution Model: Single Token
    Jobflow Execution Model: Multiple Tokens
    Stopping on Error
    Synchronous vs. Asynchronous Execution
    Logging
    Advanced Concepts
    Daemon Jobs
    Killing Jobs
    48. Jobflow Design Patterns
    Data Services
    49. Data Services Introduction
    50. Architecture
    51. Development
    Data Service Job Editor
    Endpoint Configuration
    Data Service REST Job Logic
    Anatomy of Data Service Jobs
    Input and Output Components
    HTTP Request Payload
    HTTP Request Parameters
    HTTP Headers
    HTTP Response
    Multiple Edges
    HTTP Status Code and Headers
    Execution Steps of Data Service Jobs
    Exceptions and Error Handling
    Auto-generated Documentation and Swagger/OpenAPI Definition
    Testing
    Testing Service Logic in Designer
    Testing Services Deployed on Server
    52. Use cases
    Creating REST APIs
    Custom Serialization
    Using Static File as Response
    Sending a File Generated by .rjob
    Publishing a Static File
    Using CTL2 Functions in Data Services
    Data Service that Receives a File or Text in Body Part
    Converting Graph to Data Service
    Converting Graph to Data Service in Server Projects
    Converting Graph to Data Service in Local Projects
    Publishing Data Service
    Publishing Multiple Data Services at Once
    Unpublishing Data Service
    Unpublishing Multiple Data Services at Once
    53. Example
    54. Troubleshooting
    Server Returns Error Code 404
    Server Returns Error Code 500
    Server Returns Error Code 503
    Libraries Development
    55. Libraries
    Introduction
    Usage
    Subgraphs
    Metadata
    Graphs, Jobflows
    Data Services
    Library File Structure
    Enhancing Library Look and Feel
    Parameters
    Global Library Parameters
    Connector Parameters
    Initialization Jobs (pre-generating metadata)
    User Credentials and Secrets
    Database/JDBC Connections
    OAuth2 Connections
    Library Dependencies
    Using Java Code
    Live Debugging and Testing
    Job Documentation Tips
    Documentation for Developers
    Documentation for Administrators
    Data Source Connector Development
    Data Source Connector Requirements
    Data Source Connector Best Practices
    Data Target Connector Development
    Data Target Connector Requirements
    Data Target Connector Best Practices
    Component Reference
    56. Readers
    Common Properties of Readers
    Supported File URL Formats for Readers
    Viewing Data on Readers
    Input Port Reading
    Incremental Reading
    Selecting Input Records
    Data Policy
    XML Features
    CTL Templates for Readers
    Java Interfaces for Readers
    CloverDataReader
    ComplexDataReader
    CustomJavaReader
    DatabaseReader
    DataGenerator
    DBFDataReader
    EDIFACTReader
    EmailReader
    FlatFileReader
    HadoopReader
    HL7Reader
    JavaBeanReader
    JMSReader
    JSONExtract
    JSONReader
    KafkaCommit
    KafkaReader
    LDAPReader
    MongoDBReader
    MultiLevelReader
    ParallelReader
    ParquetReader
    QuickBaseRecordReader
    QuickBaseQueryReader
    SalesforceBulkReader
    SalesforceReader
    SpreadsheetDataReader
    UniversalDataReader
    X12Reader
    XMLExtract
    XMLReader
    XMLXPathReader
    57. Writers
    Common Properties of Writers
    Supported File URL Formats for Writers
    Viewing Data on Writers
    Output Port Writing
    Appending or Overwriting
    Creating Directories
    Selecting Output Records
    Partitioning Output into Different Output Files
    Excluding Fields
    Java Interfaces for Writers
    CloverDataWriter
    CustomJavaWriter
    DatabaseWriter
    DB2BulkWriter
    DBFDataWriter
    EDIFACTWriter
    EmailSender
    FlatFileWriter
    HadoopWriter
    InformixBulkWriter
    JavaBeanWriter
    JavaMapWriter
    JMSWriter
    JSONWriter
    KafkaWriter
    LDAPWriter
    MongoDBWriter
    MSSQLBulkWriter
    MySQLBulkWriter
    OracleBulkWriter
    ParquetWriter
    PostgreSQLBulkWriter
    QuickBaseImportCSV
    QuickBaseRecordWriter
    SalesforceBulkWriter
    SalesforceWriter
    SalesforceEinsteinWriter
    SnowflakeBulkWriter
    SpreadsheetDataWriter
    StructuredDataWriter
    Trash
    UniversalDataWriter
    X12Writer
    XMLWriter
    58. Transformers
    Common Properties of Transformers
    CTL Templates for Transformers
    Java Interfaces for Transformers
    Aggregate
    Concatenate
    CustomJavaTransformer
    DataIntersection
    DataSampler
    Dedup
    Denormalizer
    ExtSort
    FastSort
    Filter
    LoadBalancingPartition
    Merge
    MetaPivot
    Normalizer
    Partition
    Pivot
    Map
    Rollup
    SimpleCopy
    SimpleGather
    SortWithinGroups
    XSLTransformer
    59. Joiners
    Common Properties of Joiners
    Join Types
    Slave Duplicates
    CTL Templates for Joiners
    Java Interfaces for Joiners
    Combine
    CrossJoin
    DBJoin
    ExtHashJoin
    ExtMergeJoin
    LookupJoin
    RelationalJoin
    60. Job Control
    Common Properties of Job Control
    Barrier
    Condition
    ExecuteGraph
    ExecuteJobflow
    ExecuteWranglerJob
    ExecuteMapReduce
    ExecuteScript
    Fail
    GetJobInput
    KillGraph
    KillJobflow
    Loop
    MonitorGraph
    MonitorJobflow
    SetJobOutput
    Sleep
    Subgraph
    Success
    TokenGather
    61. File Operations
    Common Properties of File Operations
    Common Attributes of File Operation Components
    Supported URL Formats for File Operations
    CopyFiles
    CreateFiles
    DeleteFiles
    ListFiles
    MoveFiles
    62. Data Partitioning
    Common Properties of Data Partitioning Components
    ParallelLoadBalancingPartition
    ParallelMerge
    ParallelPartition
    ParallelRepartition
    ParallelSimpleCopy
    ParallelSimpleGather
    63. Data Quality
    Common Properties of Data Quality
    EmailFilter
    ProfilerProbe
    Metrics
    List of Metrics
    Validator
    List of Rules
    64. Others
    Common Properties of Others
    CheckForeignKey
    CustomJavaComponent
    DBExecute
    HTTPConnector
    LookupTableReaderWriter
    SequenceChecker
    WebServiceClient
    65. Deprecated
    ApproximativeJoin
    InfobrightBulkWriter
    JavaExecute
    MongoDBExecute
    RunGraph
    SystemExecute
    TableauWriter
    CTL2 - CloverDX Transformation Language
    66. CTL Overview
    Basic Features of CTL
    CTL History
    67. Language Reference
    Program Structure
    Comments
    Import
    Metadata Import in CTL2
    Data Types in CTL2
    byte
    cbyte
    date
    decimal
    integer
    long
    number (double)
    string
    list
    map
    variant
    record
    Literals
    Variables
    Constants
    Dictionary in CTL2
    Operators
    Arithmetic Operators
    Relational Operators
    Logical Operators
    Assignment Operator
    Ternary Operator
    Conditional Fail Expression
    Simple Statement and Block of Statements
    Control Statements
    Conditional Statements
    Iteration Statements
    Jump Statements
    Error Handling
    Try-Catch Statement
    OnError() Functions
    Functions
    Message Function
    Conditional Fail Expression
    Accessing Data Records and Fields
    Mapping
    Parameters
    Regular Expressions
    Regex basics
    Regex for advanced users
    68. CTL Debugging
    Debug Perspective
    Importing and Exporting Breakpoints
    Exporting Breakpoints
    Importing Breakpoints
    Inspecting Variables and Expressions
    Inspect Action
    Expressions View and Watch Action
    Examples
    Basic Example
    Using Hit Count
    Conditional Breakpoint
    Detecting Changes of the Value
    69. Functions Reference
    Conversion Functions
    Date Functions
    Mathematical Functions
    String Functions
    Mapping Functions
    Container Functions
    Record Functions (Dynamic Field Access)
    Miscellaneous Functions
    Lookup Table Functions
    Sequence Functions
    Subgraph Functions
    Data Service HTTP Library Functions
    Custom CTL Functions
    List of All CTL2 Functions
    CTL2 Appendix - List of National-specific Characters
    Tutorial
    70. Creating a Transformation Graph
    Terminology
    Creating a Project
    Creating a New Data File
    Creating a Graph
    Placing Components in the Graph Editor Pane
    Connecting Components by an Edge
    Extracting Metadata from the Input File
    Assigning Metadata to the Edges
    Setting Up Readers (FlatFileReader)
    Setting Up Writers (SpreadsheetDataWriter)
    Running the Graph
    Opening the Output File
    What to do next
    71. Filtering the records
    Inserting the Filter
    Setting Up the Filter Component
    Two data streams
    72. Sorting the Records
    Adding ExtSort
    Setting Up the ExtSort Component