Normalizer
Short Description
Normalizer creates one or more output records from each single input record. Input records do not have to be sorted.
Component | Same input metadata | Sorted inputs | Inputs | Outputs | Java | CTL | Auto-propagated metadata |
---|---|---|---|---|---|---|---|
Normalizer |
- |
⨯ |
1 |
1 |
✓ |
✓ |
⨯ |
Ports
Port type | Number | Required | Description | Metadata |
---|---|---|---|---|
Input |
0 |
✓ |
For input data records |
Any1 |
Output |
0 |
✓ |
For normalized data records |
Any2 |
Metadata
Normalizer does not propagate metadata.
Normalizer does not have any metadata template.
Normalizer does not require any specific metadata fields.
Normalizer Attributes
Attribute | Req | Description | Possible values |
---|---|---|---|
Basic |
|||
Normalize |
The definition of the way how records should be normalized, written in the graph in CTL or Java. |
||
Normalize URL |
The name of an external file, including the path, containing the definition of the way how records should be normalized written in CTL or Java. |
||
Normalize class |
The name of an external class defining the way how records should be normalized. |
||
Normalize source charset |
Encoding of an external file defining the transformation. The default encoding depends on |
E.g. UTF-8 |
|
Deprecated |
|||
Error actions |
Definition of the action that should be performed when the specified transformation returns some Error code. See Return Values of Transformations. |
||
Error log |
The URL of the file to which error messages for specified Error actions should be written. If not set, they are written to Console. |
Details
Normalizer requires transformation. The transformation can be defined in CTL (see CTL Interface) or in Java (see Java Interface).
The transformation is defined using several functions. Each of them has its own purpose. The order of function calls is depicted in diagram below.
The number of calls of transform()
function depends on the result of the count()
(or countOnError()
) function.
CTL Interface
The transformation written in CTL uses a CTL template for Normalizer.
Only the functions count()
and transform()
are mandatory.
Other functions are optional.
Once you have written your transformation, you can also convert it to Java language code using the corresponding button in the upper right corner of the tab.
CTL Templates for Normalizer
CTL Template Functions | |
---|---|
boolean init() |
|
Required |
No |
Description |
Initializes the component, sets up the environment, global variables. |
Invocation |
Called before processing the first record. |
Returns |
|
integer count() |
|
Required |
yes |
Input Parameters |
none |
Returns |
The returned number defines the number of new output records that will be created by the If the |
Invocation |
Called repeatedly, once for each input record |
Description |
For each input record it generates the number of output records that will be created from this input. If If any of the input records causes the |
Example |
|
integer transform(integer idx) |
|
Required |
yes |
Input Parameters |
|
Returns |
Integer number. The number corresponds to the return value of transformation. See Return Values of Transformations. |
Invocation |
Called repeatedly, once for each output record.
The number of calls is defined by return value of function |
Description |
Creates output records. If If any part of the |
Example |
|
void clean() |
|
Required |
no |
Input Parameters |
none |
Returns |
void |
Invocation |
Called repeatedly, once for each input record. The function is called after the corresponding call(s) of |
Description |
Returns the component to the initial settings |
Example |
|
integer countOnError(string errorMessage, string stackTrace) |
|
Required |
no |
Input Parameters |
|
|
|
Returns |
The returned number defines the number of new output records that will be created by the If the |
Invocation |
Called if |
Description |
For each input record it generates the number of output records that will be created from this input. If any of the input records causes fail of the |
Example |
|
integer transformOnError(string errorMessage, string stackTrace, integer idx) |
|
Required |
no |
Input Parameters |
|
|
|
|
|
Returns |
Integer numbers. For more information, see Return Values of Transformations. |
Invocation |
Called if |
Description |
Creates output records. If any part of the The |
Example |
|
string getMessage() |
|
Required |
No |
Description |
Prints the error message specified and invoked by the user |
Invocation |
Called in any time specified by the user (called only when either |
Returns |
|
void preExecute() |
|
Required |
No |
Input parameters |
None |
Returns |
|
Description |
May be used to allocate and initialize resources required by the transform.
All resources allocated within this function should be released by the |
Invocation |
Called during each graph run before the transform is executed. |
void postExecute() |
|
Required |
No |
Input parameters |
None |
Returns |
|
Description |
Should be used to free up any resources allocated within the |
Invocation |
Called during each graph run after the entire transform was executed. |
Access to input and output fields
Input records or fields
Input records or fields are accessible within the count()
, countOnError()
, transform()
and transformOnError()
functions only.
Output records or fields
Output records or fields are accessible within the transform()
and transformOnError()
functions only.
All of the other CTL template functions allow to access neither inputs nor outputs. Remember that if you do not hold these rules, NPE will be thrown! |
Java Interface
The transformation implements methods of the RecordNormalize
interface and inherits other common methods from the Transform
interface.
See Common Java Interfaces and Public CloverDX API.
Following are the methods of RecordNormalize
interface:
-
boolean init(Properties parameters, DataRecordMetadata sourceMetadata, DataRecordMetadata targetMetadata)
Initializes normalize class/function. This method is called only once at the beginning of normalization process. Any object allocation/initialization should happen here.
-
int count(DataRecord source)
Returns the number of output records which will be created from specified input record.
-
int countOnError(Exception exception, DataRecord source)
Called only if
count(DataRecord)
throws an exception. -
int transform(DataRecord source, DataRecord target, int idx)
idx
is a sequential number of output record (starting from 0). For detailed information about return values and their meaning, see Return Values of Transformations. In Normalizer, onlyALL
,0
,SKIP
, and Error codes have some meaning. -
int transformOnError(Exception exception, DataRecord source, DataRecord target, int idx)
Called only if
transform(DataRecord, DataRecord, int)
throws an exception. -
void clean()
Finalizes current round/clean after current round - called after the transform method was called for the input record.
Examples
Converting multivalue fields to multiple records
Input records contain group name and list of users of the group. Convert records into tuples having group name and one username.
accounting | [johnsmith, elisabethtaylor] development | [georgegreen, janegreen, peterbrown]
Solution
Define the transformation using Normalize attribute.
//#CTL2
function integer count() {
return length($in.0.users);
}
function integer transform(integer idx) {
$out.0.group = $in.0.group;
$out.0.user = $in.0.users[idx];
return OK;
}
Normalizer will return following records:
accounting |johnsmith accounting |elisabethtaylor development|georgegreen development|janegreen development|peterbrown
Best Practices
If the transformation is specified in an external file (with Normalize URL), we recommend users to explicitly specify Normalize source charset.