Common Properties of Transformers
Transformers have both input and output ports. They can:
-
put together more data flows with the same metadata (Concatenate, SimpleGather and Merge);
-
remove duplicate records (Dedup);
-
filter data records (Filter and EmailFilter);
-
create samples from input records (DataSampler), sort data records (ExtSort, FastSort and SortWithinGroups);
-
multiply existing data flow (SimpleCopy);
-
split one data flow into more data flows (Partition at all, but optionally also Dedup, Filter and Map);
-
intersect two data flows (even with different metadata on inputs) (DataIntersection), aggregate data information (Aggregate);
-
and perform much more complicated transformations of data flows (Map, Denormalizer, Normalizer, Rollup and XSLTransformer).
Metadata can be propagated through some of these transformers, whereas the same is not possible in such components that transform data flows in a more complicated manner. You must have the output metadata defined prior to configuring these components.
Some of these transformers use transformations that have been described above. For detailed information about how transformation should be defined, see Defining Transformations.
-
Some Transformers can have a transformation attribute defined, it may be optional or required. For information about transformation templates for transformations written in CTL, see CTL Templates for Transformers.
-
Some Transformers can have a transformation attribute defined, it may be optional or required. For information about transformation interfaces that must be implemented in transformations written in Java, see: Java Interfaces for Transformers.
Below is an overview of all Transformers:
| Component | Same input metadata | Sorted inputs | Inputs | Outputs | Java | CTL | Auto-propagated metadata |
|---|---|---|---|---|---|---|---|
- |
⨯ |
1 |
1 |
⨯ |
⨯ |
⨯ |
|
✓ |
⨯ |
1-n |
1 |
⨯ |
⨯ |
✓ |
|
⨯ |
✓ |
2 |
3 |
✓ |
✓ |
✓ |
|
- |
⨯ |
1 |
n |
⨯ |
⨯ |
✓ |
|
- |
✓ |
1 |
1-2 |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
1 |
1 |
✓ |
✓ |
⨯ |
|
- |
⨯ |
1 |
1-n |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
1 |
1-n |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
1 |
1-2 |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
1 |
1-n |
⨯ |
⨯ |
✓ |
|
✓ |
✓ |
2-n |
1 |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
1 |
1 |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
1 |
1 |
✓ |
✓ |
⨯ |
|
- |
⨯ |
1 |
1-n |
✓ |
|||
- |
⨯ |
1 |
1 |
✓ |
✓ |
⨯ |
|
- |
⨯ |
1 |
1-n |
✓ |
✓ |
✓ |
|
- |
⨯ |
1 |
1-n |
✓ |
✓ |
⨯ |
|
- |
⨯ |
1 |
1-n |
⨯ |
⨯ |
✓ |
|
✓ |
⨯ |
1-n |
1 |
⨯ |
⨯ |
✓ |
|
- |
✓ |
1 |
1-n |
⨯ |
⨯ |
✓ |
|
- |
⨯ |
0-1 |
0-2 |
⨯ |
⨯ |
⨯ |
Partition can use either a transformation or two other attributes (Ranges or Partition key). The transformation must be defined unless one of these is specified.