Version
2. Data Manager user guide
In this part of the User’s guide, we’ll cover the Data Manager and its usage. Data Manager allows users to directly interact with data as part of the data process in CloverDX.
As an example, let’s imagine a data ingestion process. It starts with automated processes that pick-up data from a variety of sources, validate and map this data to common format before finally loading it to internal platform.
Commonly, the data validation step uncovers issues with the data that cannot be fixed automatically. There are few typical approaches for handling this kind of scenario:
-
Reject the whole source file (or batch etc.) and request the source system to fix their data and resend the fixed records.
-
Reject the data, collect it in a location (a file, a database) and notify users. Users then fix the data in some way (e.g., edit the file) and resubmit back to the process.
The first approach is more systematic as it results in the data being corrected in the source which then ensures that if a query for the same data is issues in the future, the corrected data will be returned. However, this is not always possible or feasible.
Hence the second approach is often selected. The data is sent to users in variety of means, but very commonly in an Excel spreadsheet. This allows the non-technical users to fix the data in a comfortable environment. However, this means that the data leaves the platform and is not controlled. Excel is very permissive and new errors can be introduced in the data especially when many people need to interact with it.
The Data Manager offers a powerful alternative to this approach. The records are loaded into the Data Manager which offers an easy-to-use yet powerful user interface that allows non-technical users to work on their data. At the same time, the Data Manager collects audit log of the data changes and implements an approval process to minimize the chance of errors.
Figure 77. High-level diagram of a common use case for the Data Manager - an ingestion process that reads and validates the data before sending it to the Data Manager for correction.
In this guide we’ll cover the usage of the Data Manager from multiple points of view:
-
Users editing the data can learn how to work with rows in the Data Manager in Working with data section.
-
Users managing data sets can learn how to create and manage their data sets in Managing data sets section.
Additional information required to implement a full solution using the Data Manager can be found in different parts of the CloverDX documentation:
-
Guide for CloverDX Server administrators to help them manage the Data Manager instance can be found in the Data Manager administration.
-
Details about components that can be used to interact with the data in Data Manager can be found in the documentation for DataSetReader, DataSetWriter and DataSetCommit components.