This chapter provides comprehensive guidance on the essential elements within CloverDX, which are crucial for designing and managing data transformation jobs, enabling effective data integration and processing workflows.
-
Components are the building blocks of CloverDX jobs, performing various data transformation tasks. This section covers how to add components, edit their properties, and enable or disable them based on specific conditions. You will also learn that some components have metadata templates.
-
Edges connect components and define the flow of data between them. This section explains how to create and manage edges, assign metadata, and what edge types exist. You will also learn about edge memory allocation and debugging techniques, such as using the Data Inspector or adjusting the default debug properties .
-
Metadata defines the structure of data processed within CloverDX jobs. This section introduces the Metadata Editor and its key functionalities. It covers various metadata types, including internal, external (shared), and SQL query metadata. This section also introduces the different ways to create metadata. Additionally, it discusses metadata propagation, how to change or define delimiters, or how to set up multivalue fields.
-
Connections allow CloverDX to interact with external systems and data sources. This section provides detailed information on configuring different types of connections, including database, JMS, QuickBase, Hadoop, Kafka, OAuth2, MongoDB, and Salesforce connections. Each type of connection is explained with instructions on how to set up and manage it effectively.
-
Parameters are constants or macros you define once and reuse throughout your graph configuration. This centralized management allows you to easily update values in one location without modifying them in multiple places. This section explains how to create parameters, describes the difference between internal and external (shared) parameters, and explains the importance of secure parameters to encrypt sensitive data. Additionally, it covers dynamic parameters, whose values are calculated based on a CTL formula, and environment variables, which are defined outside of CloverDX within the operating system. Additionally, it explores built-in execution parameters that can be used to tailor job behavior further within CloverDX.
-
Sequences are designed to generate a sequence of numbers, which can be used for tasks like generating unique identifiers for records. This section discusses the different types of sequences, including persistent, non-persistent, internal, and external (shared) sequences, and how to manage them.
-
Lookup tables are data structures used to map input data to corresponding values, facilitating efficient data transformation processes. Lookup tables can be accessed using CTL functions and may store data in files, databases, or cache memory. This section covers the different types of lookup tables: Simple lookup, Database lookup, Range lookup, Persistent lookup, Aspell lookup tables. These lookup tables can be internal or external (shared).
-
Notes are used to add annotations and documentation directly into CloverDX jobs. This section describes how to place, resize, edit, and format or change properties of notes, as well as how to use links and fold notes for better graph organization and readability.
-
A Dictionary in CloverDX is a data storage object associated with each job run, designed to provide simple and type-safe storage for various parameters required by the job. It is not limited only to input or output parameters but also facilitates sharing data between different components within a single job. This section guides you on creating and using dictionaries effectively in your projects.
-
Execution properties are configurable parameters that control various aspects of how a CloverDX job is executed, such as logging levels or concurrency limits. This section provides a list of available execution properties and their descriptions, and instructions on how to set them up in the Designer or Server environment.