DataSetReader reads data from data sets stored in the Data Manager.

The DataSetReader component propagates metadata on the output port. Metadata is created based on the layout of the selected data set.

If different metadata is used, the mapping from data set’s metadata to the metadata on the output port can be done via the Output mapping component attribute.

DataSetReader connects to an instance of Data Manager running on the same Server as the component and returns data from the selected data set. The component is intended for usage in “post-processing” jobs which read data from the Data Manager and load the data to the target system.

The basic pattern for reading data from Data Manager is to use DataSetReader component followed by any components implementing the logic for the data and then followed by the DataSetCommit to mark the records as fully processed.

As an example, a simple job pulling data from the Data Manager may look like this:

DataSetReader must be used together with DataSetCommit component to mark the records processed with the reader as Committed. If this is not done, the records will stay in Approved status and will never be purged from the Data Manager.

The job above first reads data from the specified data set with the DataSetReader, then loads the records to the warehouse and finally notifies the Data Manager that the records were successfully processed by setting their status to Committed with the DataSetCommit.

Note how the DataSetCommit is in phase 5 while the DataSetReader and WriteListingToDWH are both in phase 0. This two-phase approach is necessary since it is possible that the records that are read from the data set do not make it to their destination – for example, they may be rejected by an API, or the target system may be unavailable when the job runs etc.

In such cases, the records will not be marked as Committed in the data set and will be picked up again next time the job runs.