CloverDX Data Profiler
  About CloverDX Data Profiler
  Installation
    Hardware Requirements
    Software Requirements
    Licensing
    Getting CloverDX Data Profiler
  Configuration
    Run Configurations
      Log level
    Preferences
      CloverDX Data Profiler
      Results Database
  Using Profiler
    CloverDX Data Profiler GUI
    CloverDX Data Profiler essentials
    Working With Projects
    Profiling Session Step-by-step
  Profiling Jobs
    Creating Jobs
      File Profiling
      Database Table Profiling
      Excel Sheet Profiling
      Metrics
      Data Sampling
    Modifying Jobs
    Running Jobs
  Reporting Console
    Screens hierarchy
      Job List
      Job View
      Run View
      Field View
    Reporting Console as Standalone Server
      Integration with CloverDX Server
    Hardware and Software Requirements
    Installation
      Tomcat
      Websphere
      Weblogic
      JBoss
      Default Configuration
    Configuration
      Base Configuration
      Alternative ways of configuration
    Database Support
      Derby
      MSSQL
      MySQL
      Oracle
      PostgreSQL
      JNDI
    Configuring CloverDX Server
      Connection pooling
    User Accounts Settings
    Permissions
      Group-level permissions
      Public API Authorization
    Public API
      Historic Metric Values
      RSS Feed
  Profiler and CloverDX Graphs
    Data Profiler Graph Components
      ProfilerProbe
      ExecuteProfilerJob
    Profiler-Designer Element Sharing
      Sharing From Designer to Profiler
      Sharing From Profiler to Designer
  List of Metrics
  About CloverDX Data Profiler
  Installation
    Hardware Requirements
    Software Requirements
    Licensing
    Getting CloverDX Data Profiler
  Configuration
    Run Configurations
      Log level
    Preferences
      CloverDX Data Profiler
      Results Database
  Using Profiler
    CloverDX Data Profiler GUI
    CloverDX Data Profiler essentials
    Working With Projects
    Profiling Session Step-by-step
  Profiling Jobs
    Creating Jobs
      File Profiling
      Database Table Profiling
      Excel Sheet Profiling
      Metrics
      Data Sampling
    Modifying Jobs
    Running Jobs
  Reporting Console
    Screens hierarchy
      Job List
      Job View
      Run View
      Field View
    Reporting Console as Standalone Server
      Integration with CloverDX Server
    Hardware and Software Requirements
    Installation
      Tomcat
      Websphere
      Weblogic
      JBoss
      Default Configuration
    Configuration
      Base Configuration
      Alternative ways of configuration
    Database Support
      Derby
      MSSQL
      MySQL
      Oracle
      PostgreSQL
      JNDI
    Configuring CloverDX Server
      Connection pooling
    User Accounts Settings
    Permissions
      Group-level permissions
      Public API Authorization
    Public API
      Historic Metric Values
      RSS Feed
  Profiler and CloverDX Graphs
    Data Profiler Graph Components
      ProfilerProbe
      ExecuteProfilerJob
    Profiler-Designer Element Sharing
      Sharing From Designer to Profiler
      Sharing From Profiler to Designer
  List of Metrics
CloverDX Designer
  Overview
    Family of CloverDX Products
      Introduction
      CloverDX Designer
      CloverDX Server
      CloverDX Cluster
    Online Resources
  Installation
    System Requirements
      Installation
      Related Links
    Downloading
    Installing
    Upgrading
    Starting
    Activating
      Activation Using License Key
      Activation Online
    License Manager
      CloverDX License Dialog
    IBM InfoSphere MDM Plugin Installation
      Downloading
      Requirements
      Installation into Designer
        Troubleshooting
      Installation into Server
        Troubleshooting
    Optional Installation Steps
      Support for SMB 2.x and 3.x
        CloverDX Designer
        CloverDX Server
    Troubleshooting
      Windows
        Windows SmartScreen
        User Account Control
        Windows 10 Firewall
      Linux
        Designer Stops Responding
        Welcome Page not Displayed
        Hints on Edges Have no Content
      Others
        Subclipse
  Configuration
    Configuration
    Runtime Configuration
      Logging
      Master Password
      User Classpath
    CloverDX Server Integration
      Ignored Files
    Execution Monitoring
    Java Configuration
    Engine Configuration
    Refresh Operation
  Using Designer
    Designer User Interface
      Designer Panes
        Graph Editor with Palette of Components
        Navigator Pane
        Outline Pane
        Tabs Pane
        Execution Tab
      Keyboard Shortcuts
    Projects
      Types of CloverDX Projects
        CloverDX (Local) Project
        CloverDX Server Project
      Creating CloverDX Projects
        CloverDX Project
        CloverDX Server Project
        CloverDX Examples Project
      Converting CloverDX Projects
        Converting Local Project to Server Project
        Converting Server Projects to Local Project
      Structure of CloverDX Projects
        Standard Structure of All CloverDX Projects
        The .classpath File
        Workspace.prm File
      Versioning of Server Project Content
        Initial Check-Out of Project from Repository
        Adding Server Project to Version Control
        Connecting Server Project to Existing Repository
        Getting Changes from Repository
        Committing into Repository
      Working with CloverDX Server Projects
        Working Offline
        Handling Conflicts
      Project Configuration
        CloverDX Connection
        Ignored Files
    Graphs
      Creating an Empty Graph
      Creating a Simple Graph
    Execution
      Successful Graph Execution
      Run Configuration
        Main Tab
        Parameters Tab
        Refresh Tab
      Connecting to a Running Job
      Graph States
    Common Dialogs
      URL File Dialog
        Local Files
        Workspace View
        CloverDX Server
        Hadoop HDFS
        Remote Files
        Port
        Dictionary
        Filtering Files and Tips
      Edit Value Dialog
      Open Type Dialog
    Import
      Import CloverDX Projects
      Import from CloverDX Server Sandbox
      Import Graphs
      Import Metadata
        Metadata from XSD
        Metadata from DDL
    Export
      Convert Graph to Jobflow
      Convert Jobflow to Graph
      Convert Subgraph to Graph
      Export Graphs to HTML
      Export to CloverDX Server Sandbox
      Export Image
    Graph Tracking
      Changing Record Count Font Size
    Search Functionality
    Working with CloverDX Server
      CloverDX Server Project Basic Principles
      Connecting via HTTP
      Connecting via HTTPS
        Designer has its Own Certificate
        Designer does not have its Own Certificate
      Connecting via Proxy Server
  Graphs
    Components
      Adding Components
      Finding Components
      Edit Component Dialog
        Commands
        Attributes
      Enable/Disable Component
        Enabling Component
        Disabling Component
        Enabling by Graph Parameter
        Enabling by Connected Input Port
        Disable as Trash
      Passing Data Through Disabled Component
      Common Properties of Components
        Component Name
        Phases
        Component Allocation
      Specific Attribute Types
        Time Intervals
        Group Key
        Sort Key
      Metadata Templates
    Edges
      Connecting Components with Edges
      Types of Edges
      Assigning Metadata to Edges
      Colors of Edges
      Debugging Edges
        Selecting Debug Data
        Viewing Debug Data
        Turning Off Debug
      Edge Memory Allocation
    Metadata
      Records and Fields
        Record Types
        Data Types in Metadata
        Data Formats
        Locale and Locale Sensitivity
        Time Zone
        Autofilling Functions
      Metadata Types
        Internal Metadata
        External (Shared) Metadata
        Dynamic Metadata
        Reading Metadata from Special Sources
      Auto-propagated Metadata
        Priorities of Metadata
      Creating Metadata
        Extracting Metadata from a Flat File
        Extracting Metadata from an XLS(X) File
        Extracting Metadata from a Database
        Extracting Metadata from a DBase File
        Extracting Metadata from Salesforce
        Extracting Metadata from Lotus Notes
        User Defined Metadata
      Merging Existing Metadata
      Creating Database Table from Metadata and Database Connection
      Metadata Editor
        Basics of Metadata Editor
        Record Pane
        Field Name vs. Label vs. Description
        Details Pane
      Changing and Defining Delimiters
        Changing Record Delimiter
        Changing Default (Field) Delimiter
        Defining Non-Default Delimiter for a Field
      Editing Metadata in the Source Code
      Multivalue Fields
        Lists and Maps Support in Components
        Joining on Lists and Maps (Comparison Rules)
    Connections
      Database Connections
        Internal Database Connections
        External (Shared) Database Connections
        Database Connection Properties
        Encryption of Access Password
        Browsing Database and Extracting Metadata from Database Tables
        Windows Authentication on Microsoft SQL Server
        Hive Connection
        Troubleshooting
      JMS Connections
        Internal JMS Connections
        External (Shared) JMS Connections
        Edit JMS Connection Wizard
        Encrypting the Authentication Password
      QuickBase Connections
      Lotus Connections
      Hadoop Connections
        Libraries Needed for Hadoop
        Kerberos Authentication for Hadoop
      MongoDB Connections
      Salesforce Connections
        Creating Salesforce Connection
        Important Details
    Lookup Tables
      LookupTables in Cluster Environment
      Internal Lookup Tables
        Creating Internal Lookup Tables
        Externalizing Internal Lookup Tables
        Exporting Internal Lookup Tables
      External (Shared) Lookup Tables
        Creating External (Shared) Lookup Tables
        Linking External (Shared) Lookup Tables
        Internalizing External (Shared) Lookup Tables
      Types of Lookup Tables
        Simple Lookup Table
        Database Lookup Table
        Range Lookup Table
        Persistent Lookup Table
        Aspell Lookup Table
    Sequences
      Persistent Sequences
      Non Persistent Sequences
      Internal Sequences
        Creating Internal Sequences
        Externalizing Internal Sequences
        Exporting Internal Sequences
      External (Shared) Sequences
        Creating External (Shared) Sequences
        Linking External (Shared) Sequences
        Internalizing External (Shared) Sequences
      Editing a Sequence
      Sequences in Cluster Environment
    Parameters
      Internal Parameters
        Externalizing Internal Parameters
      External (Shared) Parameters
        Creating External Parameters
        Linking External Parameters
        Internalizing External (Shared) Parameters
        XML Schema of External Parameters
      Graph Parameter Editor
      Secure Graph Parameters
      Parameters with CTL Expressions (Dynamic Parameters)
      Environment Variables
      Canonicalizing File Paths
      Using Parameters
    Internal/External Graph Elements
      Internal Graph Elements
      External (Shared) Graph Elements
      Working with Graph Elements
      Advantages of External (Shared) Graph Elements
      Advantages of Internal Graph Elements
      Changing Form of Graph Elements
    Dictionary
      Creating a Dictionary
      Using a Dictionary in Graphs
        Accessing Dictionary from Readers and Writers
        Accessing Dictionary with Java
        Accessing Dictionary with CTL2
    Notes in Graphs
      Placing Notes into Graph
      Resizing Notes
      Editing Notes
      Formatted Text
      Links from Notes
      Folding Notes
      Notes Properties
      Compatibility
    Transformations
      Defining Transformations
        Components Allowing Transformation
        Java or CTL
        Internal or External Definition
        Return Values of Transformations
        Error Actions and Error Log (deprecated since 3.0)
      Transform Editor
      Common Java Interfaces
    Data Partitioning (Parallel Running)
    Data Partitioning in Cluster
      High Availability
      Scalability
        Transformation Requests
        Parallel Data Processing
      Graph Allocation Examples
      Example of Distributed Execution
        Details of the Example Transformation Design
        Scalability of the Example Transformation
      Remote Edges
  Subgraphs
    Overview
      Introduction
      Design & Execution
      Subgraphs vs. Jobflow
    Using Subgraphs
      Using Subgraphs
      Configuring Subgraphs
    Developing Subgraphs
      Wrapping
      Creating from Scratch
      Making Subgraph Configurable
      Developing and Testing Subgraphs
        Filling Required Parameters
      Metadata Propagation
    Design Patterns
      Readers
      Writers
      Transformers
      Executors
  Jobflow
    Jobflow Overview
      Introduction
        What is CloverDX Jobflow?
        Design and Execution
        Anatomy of the Jobflow Module
      Important Concepts
        Dynamic Attribute Setting
        Parameter Passing
        Pass-Through Mapping
        Execution Status Reporting
        Error Handling
        Jobflow Execution Model: Single Token
        Jobflow Execution Model: Multiple Tokens
        Stopping on Error
        Synchronous vs. Asynchronous Execution
        Logging
      Advanced Concepts
        Daemon Jobs
        Killing Jobs
    Jobflow Design Patterns
  Data Services
    Overview
    Architecture
    Development
      Data Service Job Editor
        Endpoint Configuration
        Data Service REST Job Logic
      Anatomy of Data Service Jobs
        Input and Output Components
        HTTP Request Payload
        HTTP Request Parameters
        HTTP Headers
        HTTP Response
        Multiple Edges
        HTTP Status Code and Headers
      Execution Steps of Data Service Jobs
      Exceptions and Error Handling
      Auto-generated Documentation and Swagger/OpenAPI Definition
      Testing
        Testing Service Logic in Designer
        Testing Services Deployed on Server
    Use cases
      Custom Serialization
      Sending a File Generated by .rjob
      Publishing a Static File
      Using CTL2 Functions in Data Services
      Data Service that Receives a File or Text in Body Part
      Converting Graph to Data Service
      Converting Graphs to Data Service
      Publishing Data Service
      Publishing Multiple Data Services at Once
      Unpublishing Data Service
      Unpublishing Multiple Data Services at Once
    Example
      Echo to Upper Case
    Troubleshooting
      Server Returns Error Code 404
      Server Returns Error Code 500
      Server Returns Error Code 503
  Component Reference
    Readers
      Common Properties of Readers
        Supported File URL Formats for Readers
        Viewing Data on Readers
        Input Port Reading
        Incremental Reading
        Selecting Input Records
        Data Policy
        XML Features
        CTL Templates for Readers
        Java Interfaces for Readers
      CloverDataReader
      ComplexDataReader
      CustomJavaReader
      DataGenerator
      DBFDataReader
      DBInputTable
      EmailReader
      FlatFileReader
      HadoopReader
      JavaBeanReader
      JMSReader
      JSONExtract
      JSONReader
      LDAPReader
      LotusReader
      MongoDBReader
      MultiLevelReader
      ParallelReader
      QuickBaseRecordReader
      QuickBaseQueryReader
      SalesforceBulkReader
      SalesforceReader
      SpreadsheetDataReader
      UniversalDataReader
      XMLExtract
      XMLReader
      XMLXPathReader
    Writers
      Common Properties of Writers
        Supported File URL Formats for Writers
        Viewing Data on Writers
        Output Port Writing
        Appending or Overwriting
        Creating Directories
        Selecting Output Records
        Partitioning Output into Different Output Files
        Excluding Fields
        Java Interfaces for Writers
      CloverDataWriter
      CustomJavaWriter
      DB2DataWriter
      DBFDataWriter
      DBOutputTable
      EmailSender
      FlatFileWriter
      HadoopWriter
      InfobrightDataWriter
      InformixDataWriter
      JavaBeanWriter
      JavaMapWriter
      JMSWriter
      JSONWriter
      LDAPWriter
      LotusWriter
      MongoDBWriter
      MSSQLDataWriter
      MySQLDataWriter
      OracleDataWriter
      PostgreSQLDataWriter
      QuickBaseImportCSV
      QuickBaseRecordWriter
      SalesforceBulkWriter
      SalesforceWriter
      SalesforceWaveWriter
      SpreadsheetDataWriter
      StructuredDataWriter
      TableauWriter
      Trash
      UniversalDataWriter
      XMLWriter
    Transformers
      Common Properties of Transformers
        CTL Templates for Transformers
        Java Interfaces for Transformers
      Aggregate
      Concatenate
      CustomJavaTransformer
      DataIntersection
      DataSampler
      Dedup
      Denormalizer
      ExtSort
      FastSort
      Filter
      LoadBalancingPartition
      Merge
      MetaPivot
      Normalizer
      Partition
      Pivot
      Reformat
      Rollup
      SimpleCopy
      SimpleGather
      SortWithinGroups
      XSLTransformer
    Joiners
      Common Properties of Joiners
        Join Types
        Slave Duplicates
        CTL Templates for Joiners
        Java Interfaces for Joiners
      Combine
      CrossJoin
      DBJoin
      ExtHashJoin
      ExtMergeJoin
      LookupJoin
      RelationalJoin
    Job Control
      Common Properties of Job Control
      Barrier
      Condition
      ExecuteGraph
      ExecuteJobflow
      ExecuteMapReduce
      ExecuteProfilerJob
      ExecuteScript
      Fail
      GetJobInput
      KillGraph
      KillJobflow
      Loop
      MonitorGraph
      MonitorJobflow
      SetJobOutput
      Sleep
      Subgraph
      Success
      TokenGather
    File Operations
      Common Properties of File Operations
        Supported URL Formats for File Operations
      CopyFiles
      CreateFiles
      DeleteFiles
      ListFiles
      MoveFiles
    Data Partitioning
      Common Properties of Data Partitioning Components
      ParallelLoadBalancingPartition
      ParallelMerge
      ParallelPartition
      ParallelRepartition
      ParallelSimpleCopy
      ParallelSimpleGather
    Data Quality
      Common Properties of Data Quality
      AddressDoctor 5
      EmailFilter
      ProfilerProbe
      Validator
        List of Rules
    Others
      Common Properties of Others
      CheckForeignKey
      CustomJavaComponent
      DBExecute
      HTTPConnector
      LookupTableReaderWriter
      MongoDBExecute
      RunGraph
      SequenceChecker
      SystemExecute
      WebServiceClient
    Deprecated
      ApproximativeJoin
      JavaExecute
  CTL2 - CloverDX Transformation Language
    Overview
    Language Reference
      Program Structure
      Comments
      Import
      Data Types in CTL2
        byte
        cbyte
        date
        decimal
        integer
        long
        number (double)
        string
        list
        map
        record
      Literals
      Variables
      Dictionary in CTL2
      Operators
        Arithmetic Operators
        Relational Operators
        Logical Operators
        Assignment Operator
        Ternary Operator
        Conditional Fail Expression
      Simple Statement and Block of Statements
      Control Statements
        Conditional Statements
        Iteration Statements
        Jump Statements
      Error Handling
      Functions
        Message Function
      Conditional Fail Expression
      Accessing Data Records and Fields
      Mapping
      Parameters
      Regular Expressions
    CTL Debugging
      Debug Perspective
      Importing and Exporting Breakpoints
        Exporting Breakpoints
        Importing Breakpoints
      Inspecting Variables and Expressions
        Inspect Action
        Expressions View and Watch Action
      Examples
        Basic Example
        Using Hit Count
        Conditional Breakpoint
        Detecting Changes of the Value
    Functions Reference
      Conversion Functions
      Date Functions
      Mathematical Functions
      String Functions
      Mapping Functions
      Container Functions
      Record Functions (Dynamic Field Access)
      Miscellaneous Functions
      Lookup Table Functions
      Sequence Functions
      Subgraph Functions
      Data Service HTTP Library Functions
      Custom CTL Functions
      List of All CTL2 Functions
      CTL2 Appendix - List of National-specific Characters
  Tutorial
    Creating a Transformation Graph
      Terminology
      Creating a Project
      Creating a New Data File
      Creating a Graph
      Placing Components in the Graph Editor Pane
      Connecting Components by an Edge
      Extracting Metadata from the Input File
      Assigning Metadata to the Edges
      Setting Up Readers (FlatFileReader)
      Setting Up Writers (SpreadsheetDataWriter)
      Running the Graph
      Opening the Output File
        What to do next
    Filtering the records
      Inserting the Filter
      Setting Up the Filter Component
      Two data streams
    Sorting the Records
      Adding ExtSort
      Setting Up the ExtSort Component
    Processing Speedup with Parallelization
    Debugging the Java Transformation
      Debugging the Java Transformation in Local Projects
      Debugging the Java Transformation in Server Projects
  Custom Component
    Creating a Custom Component
      Component Java Definition
      Plugin.xml
    Integrating a Custom Component in CloverDX Designer
      Plug-in Project
      Component XML Definition
      Plug-in Dependencies
      Plug-in Extensions
      Import Requisities
      Plug-in Building
      Troubleshooting
    Tips for Advanced
  Overview
    Family of CloverDX Products
      Introduction
      CloverDX Designer
      CloverDX Server
      CloverDX Cluster
    Online Resources
  Installation
    System Requirements
      Installation
      Related Links
    Downloading
    Installing
    Upgrading
    Starting
    Activating
      Activation Using License Key
      Activation Online
    License Manager
      CloverDX License Dialog
    IBM InfoSphere MDM Plugin Installation
      Downloading
      Requirements
      Installation into Designer
        Troubleshooting
      Installation into Server
        Troubleshooting
    Optional Installation Steps
      Support for SMB 2.x and 3.x
        CloverDX Designer
        CloverDX Server
    Troubleshooting
      Windows
        Windows SmartScreen
        User Account Control
        Windows 10 Firewall
      Linux
        Designer Stops Responding
        Welcome Page not Displayed
        Hints on Edges Have no Content
      Others
        Subclipse
  Configuration
    Configuration
    Runtime Configuration
      Logging
      Master Password
      User Classpath
    CloverDX Server Integration
      Ignored Files
    Execution Monitoring
    Java Configuration
    Engine Configuration
    Refresh Operation
  Using Designer
    Designer User Interface
      Designer Panes
        Graph Editor with Palette of Components
        Navigator Pane
        Outline Pane
        Tabs Pane
        Execution Tab
      Keyboard Shortcuts
    Projects
      Types of CloverDX Projects
        CloverDX (Local) Project
        CloverDX Server Project
      Creating CloverDX Projects
        CloverDX Project
        CloverDX Server Project
        CloverDX Examples Project
      Converting CloverDX Projects
        Converting Local Project to Server Project
        Converting Server Projects to Local Project
      Structure of CloverDX Projects
        Standard Structure of All CloverDX Projects
        The .classpath File
        Workspace.prm File
      Versioning of Server Project Content
        Initial Check-Out of Project from Repository
        Adding Server Project to Version Control
        Connecting Server Project to Existing Repository
        Getting Changes from Repository
        Committing into Repository
      Working with CloverDX Server Projects
        Working Offline
        Handling Conflicts
      Project Configuration
        CloverDX Connection
        Ignored Files
    Graphs
      Creating an Empty Graph
      Creating a Simple Graph
    Execution
      Successful Graph Execution
      Run Configuration
        Main Tab
        Parameters Tab
        Refresh Tab
      Connecting to a Running Job
      Graph States
    Common Dialogs
      URL File Dialog
        Local Files
        Workspace View
        CloverDX Server
        Hadoop HDFS
        Remote Files
        Port
        Dictionary
        Filtering Files and Tips
      Edit Value Dialog
      Open Type Dialog
    Import
      Import CloverDX Projects
      Import from CloverDX Server Sandbox
      Import Graphs
      Import Metadata
        Metadata from XSD
        Metadata from DDL
    Export
      Convert Graph to Jobflow
      Convert Jobflow to Graph
      Convert Subgraph to Graph
      Export Graphs to HTML
      Export to CloverDX Server Sandbox
      Export Image
    Graph Tracking
      Changing Record Count Font Size
    Search Functionality
    Working with CloverDX Server
      CloverDX Server Project Basic Principles
      Connecting via HTTP
      Connecting via HTTPS
        Designer has its Own Certificate
        Designer does not have its Own Certificate
      Connecting via Proxy Server
  Graphs
    Components
      Adding Components
      Finding Components
      Edit Component Dialog
        Commands
        Attributes
      Enable/Disable Component
        Enabling Component
        Disabling Component
        Enabling by Graph Parameter
        Enabling by Connected Input Port
        Disable as Trash
      Passing Data Through Disabled Component
      Common Properties of Components
        Component Name
        Phases
        Component Allocation
      Specific Attribute Types
        Time Intervals
        Group Key
        Sort Key
      Metadata Templates
    Edges
      Connecting Components with Edges
      Types of Edges
      Assigning Metadata to Edges
      Colors of Edges
      Debugging Edges
        Selecting Debug Data
        Viewing Debug Data
        Turning Off Debug
      Edge Memory Allocation
    Metadata
      Records and Fields
        Record Types
        Data Types in Metadata
        Data Formats
        Locale and Locale Sensitivity
        Time Zone
        Autofilling Functions
      Metadata Types
        Internal Metadata
        External (Shared) Metadata
        Dynamic Metadata
        Reading Metadata from Special Sources
      Auto-propagated Metadata
        Priorities of Metadata
      Creating Metadata
        Extracting Metadata from a Flat File
        Extracting Metadata from an XLS(X) File
        Extracting Metadata from a Database
        Extracting Metadata from a DBase File
        Extracting Metadata from Salesforce
        Extracting Metadata from Lotus Notes
        User Defined Metadata
      Merging Existing Metadata
      Creating Database Table from Metadata and Database Connection
      Metadata Editor
        Basics of Metadata Editor
        Record Pane
        Field Name vs. Label vs. Description
        Details Pane
      Changing and Defining Delimiters
        Changing Record Delimiter
        Changing Default (Field) Delimiter
        Defining Non-Default Delimiter for a Field
      Editing Metadata in the Source Code
      Multivalue Fields
        Lists and Maps Support in Components
        Joining on Lists and Maps (Comparison Rules)
    Connections
      Database Connections
        Internal Database Connections
        External (Shared) Database Connections
        Database Connection Properties
        Encryption of Access Password
        Browsing Database and Extracting Metadata from Database Tables
        Windows Authentication on Microsoft SQL Server
        Hive Connection
        Troubleshooting
      JMS Connections
        Internal JMS Connections
        External (Shared) JMS Connections
        Edit JMS Connection Wizard
        Encrypting the Authentication Password
      QuickBase Connections
      Lotus Connections
      Hadoop Connections
        Libraries Needed for Hadoop
        Kerberos Authentication for Hadoop
      MongoDB Connections
      Salesforce Connections
        Creating Salesforce Connection
        Important Details
    Lookup Tables
      LookupTables in Cluster Environment
      Internal Lookup Tables
        Creating Internal Lookup Tables
        Externalizing Internal Lookup Tables
        Exporting Internal Lookup Tables
      External (Shared) Lookup Tables
        Creating External (Shared) Lookup Tables
        Linking External (Shared) Lookup Tables
        Internalizing External (Shared) Lookup Tables
      Types of Lookup Tables
        Simple Lookup Table
        Database Lookup Table
        Range Lookup Table
        Persistent Lookup Table
        Aspell Lookup Table
    Sequences
      Persistent Sequences
      Non Persistent Sequences
      Internal Sequences
        Creating Internal Sequences
        Externalizing Internal Sequences
        Exporting Internal Sequences
      External (Shared) Sequences
        Creating External (Shared) Sequences
        Linking External (Shared) Sequences
        Internalizing External (Shared) Sequences
      Editing a Sequence
      Sequences in Cluster Environment
    Parameters
      Internal Parameters
        Externalizing Internal Parameters
      External (Shared) Parameters
        Creating External Parameters
        Linking External Parameters
        Internalizing External (Shared) Parameters
        XML Schema of External Parameters
      Graph Parameter Editor
      Secure Graph Parameters
      Parameters with CTL Expressions (Dynamic Parameters)
      Environment Variables
      Canonicalizing File Paths
      Using Parameters
    Internal/External Graph Elements
      Internal Graph Elements
      External (Shared) Graph Elements
      Working with Graph Elements
      Advantages of External (Shared) Graph Elements
      Advantages of Internal Graph Elements
      Changing Form of Graph Elements
    Dictionary
      Creating a Dictionary
      Using a Dictionary in Graphs
        Accessing Dictionary from Readers and Writers
        Accessing Dictionary with Java
        Accessing Dictionary with CTL2
    Notes in Graphs
      Placing Notes into Graph
      Resizing Notes
      Editing Notes
      Formatted Text
      Links from Notes
      Folding Notes
      Notes Properties
      Compatibility
    Transformations
      Defining Transformations
        Components Allowing Transformation
        Java or CTL
        Internal or External Definition
        Return Values of Transformations
        Error Actions and Error Log (deprecated since 3.0)
      Transform Editor
      Common Java Interfaces
    Data Partitioning (Parallel Running)
    Data Partitioning in Cluster
      High Availability
      Scalability
        Transformation Requests
        Parallel Data Processing
      Graph Allocation Examples
      Example of Distributed Execution
        Details of the Example Transformation Design
        Scalability of the Example Transformation
      Remote Edges
  Subgraphs
    Overview
      Introduction
      Design & Execution
      Subgraphs vs. Jobflow
    Using Subgraphs
      Using Subgraphs
      Configuring Subgraphs
    Developing Subgraphs
      Wrapping
      Creating from Scratch
      Making Subgraph Configurable
      Developing and Testing Subgraphs
        Filling Required Parameters
      Metadata Propagation
    Design Patterns
      Readers
      Writers
      Transformers
      Executors
  Jobflow
    Jobflow Overview
      Introduction
        What is CloverDX Jobflow?
        Design and Execution
        Anatomy of the Jobflow Module
      Important Concepts
        Dynamic Attribute Setting
        Parameter Passing
        Pass-Through Mapping
        Execution Status Reporting
        Error Handling
        Jobflow Execution Model: Single Token
        Jobflow Execution Model: Multiple Tokens
        Stopping on Error
        Synchronous vs. Asynchronous Execution
        Logging
      Advanced Concepts
        Daemon Jobs
        Killing Jobs
    Jobflow Design Patterns
  Data Services
    Overview
    Architecture
    Development
      Data Service Job Editor
        Endpoint Configuration
        Data Service REST Job Logic
      Anatomy of Data Service Jobs
        Input and Output Components
        HTTP Request Payload
        HTTP Request Parameters
        HTTP Headers
        HTTP Response
        Multiple Edges
        HTTP Status Code and Headers
      Execution Steps of Data Service Jobs
      Exceptions and Error Handling
      Auto-generated Documentation and Swagger/OpenAPI Definition
      Testing
        Testing Service Logic in Designer
        Testing Services Deployed on Server
    Use cases
      Custom Serialization
      Sending a File Generated by .rjob
      Publishing a Static File
      Using CTL2 Functions in Data Services
      Data Service that Receives a File or Text in Body Part
      Converting Graph to Data Service
      Converting Graphs to Data Service
      Publishing Data Service
      Publishing Multiple Data Services at Once
      Unpublishing Data Service
      Unpublishing Multiple Data Services at Once
    Example
      Echo to Upper Case
    Troubleshooting
      Server Returns Error Code 404
      Server Returns Error Code 500
      Server Returns Error Code 503
  Component Reference
    Readers
      Common Properties of Readers
        Supported File URL Formats for Readers
        Viewing Data on Readers
        Input Port Reading
        Incremental Reading
        Selecting Input Records
        Data Policy
        XML Features
        CTL Templates for Readers
        Java Interfaces for Readers
      CloverDataReader
      ComplexDataReader
      CustomJavaReader
      DataGenerator
      DBFDataReader
      DBInputTable
      EmailReader
      FlatFileReader
      HadoopReader
      JavaBeanReader
      JMSReader
      JSONExtract
      JSONReader
      LDAPReader
      LotusReader
      MongoDBReader
      MultiLevelReader
      ParallelReader
      QuickBaseRecordReader
      QuickBaseQueryReader
      SalesforceBulkReader
      SalesforceReader
      SpreadsheetDataReader
      UniversalDataReader
      XMLExtract
      XMLReader
      XMLXPathReader
    Writers
      Common Properties of Writers
        Supported File URL Formats for Writers
        Viewing Data on Writers
        Output Port Writing
        Appending or Overwriting
        Creating Directories
        Selecting Output Records
        Partitioning Output into Different Output Files
        Excluding Fields
        Java Interfaces for Writers
      CloverDataWriter
      CustomJavaWriter
      DB2DataWriter
      DBFDataWriter
      DBOutputTable
      EmailSender
      FlatFileWriter
      HadoopWriter
      InfobrightDataWriter
      InformixDataWriter
      JavaBeanWriter
      JavaMapWriter
      JMSWriter
      JSONWriter
      LDAPWriter
      LotusWriter
      MongoDBWriter
      MSSQLDataWriter
      MySQLDataWriter
      OracleDataWriter
      PostgreSQLDataWriter
      QuickBaseImportCSV
      QuickBaseRecordWriter
      SalesforceBulkWriter
      SalesforceWriter
      SalesforceWaveWriter
      SpreadsheetDataWriter
      StructuredDataWriter
      TableauWriter
      Trash
      UniversalDataWriter
      XMLWriter
    Transformers
      Common Properties of Transformers
        CTL Templates for Transformers
        Java Interfaces for Transformers
      Aggregate
      Concatenate
      CustomJavaTransformer
      DataIntersection
      DataSampler
      Dedup
      Denormalizer
      ExtSort
      FastSort
      Filter
      LoadBalancingPartition
      Merge
      MetaPivot
      Normalizer
      Partition
      Pivot
      Reformat
      Rollup
      SimpleCopy
      SimpleGather
      SortWithinGroups
      XSLTransformer
    Joiners
      Common Properties of Joiners
        Join Types
        Slave Duplicates
        CTL Templates for Joiners
        Java Interfaces for Joiners
      Combine
      CrossJoin
      DBJoin
      ExtHashJoin
      ExtMergeJoin
      LookupJoin
      RelationalJoin
    Job Control
      Common Properties of Job Control
      Barrier
      Condition
      ExecuteGraph
      ExecuteJobflow
      ExecuteMapReduce
      ExecuteProfilerJob
      ExecuteScript
      Fail
      GetJobInput
      KillGraph
      KillJobflow
      Loop
      MonitorGraph
      MonitorJobflow
      SetJobOutput
      Sleep
      Subgraph
      Success
      TokenGather
    File Operations
      Common Properties of File Operations
        Supported URL Formats for File Operations
      CopyFiles
      CreateFiles
      DeleteFiles
      ListFiles
      MoveFiles
    Data Partitioning
      Common Properties of Data Partitioning Components
      ParallelLoadBalancingPartition
      ParallelMerge
      ParallelPartition
      ParallelRepartition
      ParallelSimpleCopy
      ParallelSimpleGather
    Data Quality
      Common Properties of Data Quality
      AddressDoctor 5
      EmailFilter
      ProfilerProbe
      Validator
        List of Rules
    Others
      Common Properties of Others
      CheckForeignKey
      CustomJavaComponent
      DBExecute
      HTTPConnector
      LookupTableReaderWriter
      MongoDBExecute
      RunGraph
      SequenceChecker
      SystemExecute
      WebServiceClient
    Deprecated
      ApproximativeJoin
      JavaExecute
  CTL2 - CloverDX Transformation Language
    Overview
    Language Reference
      Program Structure
      Comments
      Import
      Data Types in CTL2
        byte
        cbyte
        date
        decimal
        integer
        long
        number (double)
        string
        list
        map
        record
      Literals
      Variables
      Dictionary in CTL2
      Operators
        Arithmetic Operators
        Relational Operators
        Logical Operators
        Assignment Operator
        Ternary Operator
        Conditional Fail Expression
      Simple Statement and Block of Statements
      Control Statements
        Conditional Statements
        Iteration Statements
        Jump Statements
      Error Handling
      Functions
        Message Function
      Conditional Fail Expression
      Accessing Data Records and Fields
      Mapping
      Parameters
      Regular Expressions
    CTL Debugging
      Debug Perspective
      Importing and Exporting Breakpoints
        Exporting Breakpoints
        Importing Breakpoints
      Inspecting Variables and Expressions
        Inspect Action
        Expressions View and Watch Action
      Examples
        Basic Example
        Using Hit Count
        Conditional Breakpoint
        Detecting Changes of the Value
    Functions Reference
      Conversion Functions
      Date Functions
      Mathematical Functions
      String Functions
      Mapping Functions
      Container Functions
      Record Functions (Dynamic Field Access)
      Miscellaneous Functions
      Lookup Table Functions
      Sequence Functions
      Subgraph Functions
      Data Service HTTP Library Functions
      Custom CTL Functions
      List of All CTL2 Functions
      CTL2 Appendix - List of National-specific Characters
  Tutorial
    Creating a Transformation Graph
      Terminology
      Creating a Project
      Creating a New Data File
      Creating a Graph
      Placing Components in the Graph Editor Pane
      Connecting Components by an Edge
      Extracting Metadata from the Input File
      Assigning Metadata to the Edges
      Setting Up Readers (FlatFileReader)
      Setting Up Writers (SpreadsheetDataWriter)
      Running the Graph
      Opening the Output File
        What to do next
    Filtering the records
      Inserting the Filter
      Setting Up the Filter Component
      Two data streams
    Sorting the Records
      Adding ExtSort
      Setting Up the ExtSort Component
    Processing Speedup with Parallelization
    Debugging the Java Transformation
      Debugging the Java Transformation in Local Projects
      Debugging the Java Transformation in Server Projects
  Custom Component
    Creating a Custom Component
      Component Java Definition
      Plugin.xml
    Integrating a Custom Component in CloverDX Designer
      Plug-in Project
      Component XML Definition
      Plug-in Dependencies
      Plug-in Extensions
      Import Requisities
      Plug-in Building
      Troubleshooting
    Tips for Advanced
CloverDX Server
  Introduction
    What is CloverDX Server?
    CloverDX Server Architecture
    CloverDX Core
    CloverDX Worker
    CloverDX Cluster
  Installation
    System Requirements
    Installing
      Evaluation Server
        Installation
      Production Server
        Apache Tomcat
        IBM WebSphere
        JBoss Enterprise Application Platform
        Oracle WebLogic Server
      Activation
        CloverDX Server Activation Alternatives
      IBM InfoSphere MDM Plugin Installation
        Installation into Server
    Postinstallation Configuration
      Memory Settings
      Maximum Number of Open Files
      Maximum Number of Processes or Threads
      Firewall Exceptions
      Garbage Collector for Worker
    Optional Installation Steps
      Adding Libraries to the Worker's Classpath
      Worker Support for SMB 2.x and 3.x
    Troubleshooting
  Configuration
    Introduction
    Configuration Sources
      Configuration File on Specified Location
      Alternative Configuration Sources
      Priorities of Configuration Sources
      Specifying the Path to the Configuration File
    Setup
      Before You Start
      Using Setup
        Configuration File
        License
        Database
        Worker
        Sandboxes
        Encryption
        E-Mail
        LDAP
        Cluster
    System Database Configuration
      Embedded Apache Derby
      MySQL
        Creating Database
        CloverDX Server Setup
      DB2
        Creating Database
        CloverDX Server Setup
        DB2 on AS/400
      Oracle
        Creating Database
        CloverDX Server Setup
      Microsoft SQL Server
        Creating Database
        CloverDX Server Setup
      PostgreSQL
        Creating Database
        CloverDX Server Setup
      JNDI Configuration and Encryption
        JNDI DB Datasource
        JNDI Datasource Troubleshooting
        Encrypted JNDI
    List of Configuration Properties
      General Configuration Properties
      Worker - Configuration Properties
      Worker - JNDI Properties
        JDBC Datasources
        JMS Connections
      Worker - SSL Properties
      Job Execution Properties
      List of all properties
    Secure Configuration Properties
    Logging
  Administration
    Monitoring
      Standalone Server Detail
      Cluster Overview
      Node Detail
      Server Logs
      Using the Monitoring
        Restarting the Worker
        Showing Worker's Command Line Arguments
        Suspending the Server
        Resuming the Server
        Displaying List of Threads of the Server Core
    Temp Space Management
      Overview
      Management
    Secure Parameters
    Users and Groups
      LDAP Authentication
      Users
      Groups
      User Lockout
    Sandboxes - Server Side Job Files
      Referencing Files from the Graph or Jobflow
      Sandbox Content Security and Permissions
      Sandbox Content and Options
        Download sandbox as ZIP
        Upload ZIP to sandbox
        Download file in ZIP
        Delete Sandbox
      Job Config Properties
      WebDAV Access to Sandboxes
        WebDAV Clients
        WebDAV Authentication/Authorization
    Server Configuration Migration
      Server Configuration Export
      Server Configuration Import
    Upgrading Server to Newer Version
    Diagnostics
    Troubleshooting Worker
  Using Graphs
    Graph/Jobflow Parameters
      Parameters by Execution Type
      Adding Another Graph Parameters
    Tasks
      Send an Email
        Placeholders
      Execute Shell Command
      Start a Graph
      Start a Jobflow
      Start a Profiler Job
      Abort job
      Archive Records
      Send a JMS Message
      Execute Groovy Code
    Manual Task Execution
    Scheduling
      Timetable Setting
      Allocations of Scheduled Task on Nodes
      Scheduling the Tasks - Examples
        Start a graph at specific time
        Start a Jobflow once an hour
        Complex Scheduling
    Viewing Job Runs - Execution History
      Filtering and ordering
      Tracking
      Log File
    Listeners
      Graph Event Listeners
        Graph Events
        Listener
        Tasks
        Use Cases
      Jobflow Event Listeners
        Jobflow Events
        Listener
        Tasks
      JMS Message Listeners
      Universal Event Listeners
        Evaluation Criteria
      File Event Listeners (remote and local)
        Cluster environment
        Supported filesystems and protocols
        Observed file
        File Events
        Check Interval, Task and Use Cases
        Howtos
      Task Failure Listeners
        Task Choice
        Task Failed E-mail Template
    Recommendations for Transformations Developers
    Extensibility - CloverDX Engine Plugins
    Troubleshooting
  API
    Simple HTTP API
    JMX mBean
      JMX Configuration
      Operations
    SOAP WebService API
      SOAP WS Client
      SOAP WS API Authentication/Authorization
    Data Services
      Overview
      User Interface
        Endpoints
        List of Data Services
        Detail
        Testing and Documentation
        State and History
        Alerts and Notification
        Configuration
        Catalog of Services
        Built-in Data Service Examples
        HTTPS Connectors
      Using Data Services
        Deploying Data Service
        Publishing and Unpublishing Data Service from Sandbox
        Publishing Data Service Examples
        Changing Data Service to Anonymous
        Running Data Service on HTTPS
        Running Data Service on HTTPS on Cluster
        Monitoring Data Service
        Testing Data Service
        Performance Tuning
        Exporting Data Service Configuration
        Importing Data Service Configuration
        Avoiding Premature Marking of Data Service as Failing
        Looking up Particular Data Service
        Resetting State of Failing Data Service Endpoint
      Custom HTTP Headers
      Data Services on Cluster
  Cluster
    Sandboxes in Cluster
      Using a Sandbox Resource as a Component Data Source
      Remote Edges
    Cluster Configuration
      Mandatory Cluster Properties
      Optional Cluster Properties
      Example of 2 Node Cluster Configuration
        Basic 2-nodes Cluster Configuration
        2-nodes Cluster with Proxied Access to Database
        2-nodes Cluster with Load Balancer
      Example of 3 Node Cluster Configuration
        Basic 3-nodes Cluster Configuration
      Jobs Load Balancing Properties
      Running More Clusters
    Recommendations for Cluster Deployment
    Troubleshooting
      NodeA Cannot Establish HTTP Connection to NodeB
      NodeA Cannot Establish TCP Connection (Port 7800 by Default) to NodeB
      NodeB is Killed or It Cannot Connect to the Database
      Auto-Resuming in Unreliable Network
      Long-Term Network Malfunction May Cause Jobs to Hang on
  Security
    Security Recommendations for CloverDX Server
  Introduction
    What is CloverDX Server?
    CloverDX Server Architecture
    CloverDX Core
    CloverDX Worker
    CloverDX Cluster
  Installation
    System Requirements
    Installing
      Evaluation Server
        Installation
      Production Server
        Apache Tomcat
        IBM WebSphere
        JBoss Enterprise Application Platform
        Oracle WebLogic Server
      Activation
        CloverDX Server Activation Alternatives
      IBM InfoSphere MDM Plugin Installation
        Installation into Server
    Postinstallation Configuration
      Memory Settings
      Maximum Number of Open Files
      Maximum Number of Processes or Threads
      Firewall Exceptions
      Garbage Collector for Worker
    Optional Installation Steps
      Adding Libraries to the Worker's Classpath
      Worker Support for SMB 2.x and 3.x
    Troubleshooting
  Configuration
    Introduction
    Configuration Sources
      Configuration File on Specified Location
      Alternative Configuration Sources
      Priorities of Configuration Sources
      Specifying the Path to the Configuration File
    Setup
      Before You Start
      Using Setup
        Configuration File
        License
        Database
        Worker
        Sandboxes
        Encryption
        E-Mail
        LDAP
        Cluster
    System Database Configuration
      Embedded Apache Derby
      MySQL
        Creating Database
        CloverDX Server Setup
      DB2
        Creating Database
        CloverDX Server Setup
        DB2 on AS/400
      Oracle
        Creating Database
        CloverDX Server Setup
      Microsoft SQL Server
        Creating Database
        CloverDX Server Setup
      PostgreSQL
        Creating Database
        CloverDX Server Setup
      JNDI Configuration and Encryption
        JNDI DB Datasource
        JNDI Datasource Troubleshooting
        Encrypted JNDI
    List of Configuration Properties
      General Configuration Properties
      Worker - Configuration Properties
      Worker - JNDI Properties
        JDBC Datasources
        JMS Connections
      Worker - SSL Properties
      Job Execution Properties
      List of all properties
    Secure Configuration Properties
    Logging
  Administration
    Monitoring
      Standalone Server Detail
      Cluster Overview
      Node Detail
      Server Logs
      Using the Monitoring
        Restarting the Worker
        Showing Worker's Command Line Arguments
        Suspending the Server
        Resuming the Server
        Displaying List of Threads of the Server Core
    Temp Space Management
      Overview
      Management
    Secure Parameters
    Users and Groups
      LDAP Authentication
      Users
      Groups
      User Lockout
    Sandboxes - Server Side Job Files
      Referencing Files from the Graph or Jobflow
      Sandbox Content Security and Permissions
      Sandbox Content and Options
        Download sandbox as ZIP
        Upload ZIP to sandbox
        Download file in ZIP
        Delete Sandbox
      Job Config Properties
      WebDAV Access to Sandboxes
        WebDAV Clients
        WebDAV Authentication/Authorization
    Server Configuration Migration
      Server Configuration Export
      Server Configuration Import
    Upgrading Server to Newer Version
    Diagnostics
    Troubleshooting Worker
  Using Graphs
    Graph/Jobflow Parameters
      Parameters by Execution Type
      Adding Another Graph Parameters
    Tasks
      Send an Email
        Placeholders
      Execute Shell Command
      Start a Graph
      Start a Jobflow
      Start a Profiler Job
      Abort job
      Archive Records
      Send a JMS Message
      Execute Groovy Code
    Manual Task Execution
    Scheduling
      Timetable Setting
      Allocations of Scheduled Task on Nodes
      Scheduling the Tasks - Examples
        Start a graph at specific time
        Start a Jobflow once an hour
        Complex Scheduling
    Viewing Job Runs - Execution History
      Filtering and ordering
      Tracking
      Log File
    Listeners
      Graph Event Listeners
        Graph Events
        Listener
        Tasks
        Use Cases
      Jobflow Event Listeners
        Jobflow Events
        Listener
        Tasks
      JMS Message Listeners
      Universal Event Listeners
        Evaluation Criteria
      File Event Listeners (remote and local)
        Cluster environment
        Supported filesystems and protocols
        Observed file
        File Events
        Check Interval, Task and Use Cases
        Howtos
      Task Failure Listeners
        Task Choice
        Task Failed E-mail Template
    Recommendations for Transformations Developers
    Extensibility - CloverDX Engine Plugins
    Troubleshooting
  API
    Simple HTTP API
    JMX mBean
      JMX Configuration
      Operations
    SOAP WebService API
      SOAP WS Client
      SOAP WS API Authentication/Authorization
    Data Services
      Overview
      User Interface
        Endpoints
        List of Data Services
        Detail
        Testing and Documentation
        State and History
        Alerts and Notification
        Configuration
        Catalog of Services
        Built-in Data Service Examples
        HTTPS Connectors
      Using Data Services
        Deploying Data Service
        Publishing and Unpublishing Data Service from Sandbox
        Publishing Data Service Examples
        Changing Data Service to Anonymous
        Running Data Service on HTTPS
        Running Data Service on HTTPS on Cluster
        Monitoring Data Service
        Testing Data Service
        Performance Tuning
        Exporting Data Service Configuration
        Importing Data Service Configuration
        Avoiding Premature Marking of Data Service as Failing
        Looking up Particular Data Service
        Resetting State of Failing Data Service Endpoint
      Custom HTTP Headers
      Data Services on Cluster
  Cluster
    Sandboxes in Cluster
      Using a Sandbox Resource as a Component Data Source
      Remote Edges
    Cluster Configuration
      Mandatory Cluster Properties
      Optional Cluster Properties
      Example of 2 Node Cluster Configuration
        Basic 2-nodes Cluster Configuration
        2-nodes Cluster with Proxied Access to Database
        2-nodes Cluster with Load Balancer
      Example of 3 Node Cluster Configuration
        Basic 3-nodes Cluster Configuration
      Jobs Load Balancing Properties
      Running More Clusters
    Recommendations for Cluster Deployment
    Troubleshooting
      NodeA Cannot Establish HTTP Connection to NodeB
      NodeA Cannot Establish TCP Connection (Port 7800 by Default) to NodeB
      NodeB is Killed or It Cannot Connect to the Database
      Auto-Resuming in Unreliable Network
      Long-Term Network Malfunction May Cause Jobs to Hang on
  Security
    Security Recommendations for CloverDX Server