Version

    Normalizer

    Short Description
    Ports
    Metadata
    Normalizer Attributes
    Details
    CTL Interface
    Java Interface
    Examples
    Best Practices
    See also

    Short Description

    Normalizer creates one or more output records from each single input record. Input records do not have to be sorted.

    ComponentSame input metadataSorted inputsInputsOutputsJavaCTLAuto-propagated metadata
    Normalizer-
    no
    11
    yes
    yes
    no

    Ports

    Port typeNumberRequiredDescriptionMetadata
    Input0
    yes
    For input data recordsAny1
    Output0
    yes
    For normalized data recordsAny2

    Metadata

    Normalizer does not propagate metadata.

    Normalizer does not have any metadata template.

    Normalizer does not require any specific metadata fields.

    Normalizer Attributes

    AttributeReqDescriptionPossible values
    Basic
    Normalize[1]The definition of the way how records should be normalized, written in the graph in CTL or Java. 
    Normalize URL[1]

    The name of an external file, including the path, containing the definition of the way how records should be normalized written in CTL or Java.

     
    Normalize class[1]The name of an external class defining the way how records should be normalized. 
    Normalize source charset 

    Encoding of an external file defining the transformation.

    The default encoding depends on DEFAULT_SOURCE_CODE_CHARSET in defaultProperties.

    E.g. UTF-8
    Deprecated
    Error actions 

    Definition of the action that should be performed when the specified transformation returns some Error code. See Return Values of Transformations.

     
    Error log 

    The URL of the file to which error messages for specified Error actions should be written. If not set, they are written to Console.

     

    [1]  One of these must specified.

    Details

    Normalizer requires transformation. The transformation can be defined in CTL (see CTL Interface) or in Java (see Java Interface).

    The transformation is defined using several functions. Each of them has its own purpose. The order of function calls is depicted in diagram below.

    Normalizer code workflow

    Figure 57.3. Normalizer code workflow


    The number of calls of transform() function depends on the result of the count() (or countOnError()) function.

    CTL Interface

    CTL Templates for Normalizer
    Access to input and output fields

    The transformation written in CTL uses a CTL template for Normalizer. Only the functions count() and transform() are mandatory. Other functions are optional.

    Once you have written your transformation, you can also convert it to Java language code using the corresponding button in the upper right corner of the tab.

    CTL Templates for Normalizer

    Table 57.4. Functions in Normalizer

    CTL Template Functions
    boolean init()
    RequiredNo
    DescriptionInitializes the component, sets up the environment, global variables.
    InvocationCalled before processing the first record.
    Returnstrue | false (in case of false graph fails)
    integer count()
    Requiredyes
    Input Parametersnone
    Returns

    The returned number defines the number of new output records that will be created by the transform() function.

    If the count() function returns 0, the subsequent call of transform() is skipped (transform() is called zero times).

    InvocationCalled repeatedly, once for each input record
    Description

    For each input record it generates the number of output records that will be created from this input.

    If count() fails and user has not defined any countOnError(), the whole graph will fail.

    If any of the input records causes the count() function to fail, and if user has defined another function (countOnError()), processing continues in this countOnError() at the place where count() failed. The countOnError() function gets the information gathered by count() that was received from previously successfully processed input records. Also the error message and stack trace are passed to countOnError().

    Example
    function integer count() {
       customers = split($in.0.customers,"-");
       return length(customers);
    }
    integer transform(integer idx)
    Requiredyes
    Input Parameters

    integer idx integer numbers from 0 to count-1 (Here count is the number returned by the count() function.)

    Returns

    Integer number. The number corresponds to the return value of transformation. See Return Values of Transformations.

    Invocation

    Called repeatedly, once for each output record. The number of calls is defined by return value of function count().

    Description

    Creates output records.

    If transform() fails and the user has not defined any transformOnError(), the whole graph will fail.

    If any part of the transform() function for some output record causes fail of the transform() function, and if user has defined another function (transformOnError()), processing continues in this transformOnError() at the place where transform() failed. The transformOnError() function gets the information gathered by transform() that was received from previously successfully processed code. Also the error message and stack trace are passed to transformOnError().

    Example
    function integer transform(integer idx) {
       myString = customers[idx];
       $out.0.OneCustomer = str2integer(myString);
       $out.0.RecordNo = $in.0.recordNo;
       $out.0.OrderWithinRecord = idx;	
       return OK;
    }
    void clean()
    Requiredno
    Input Parametersnone
    Returnsvoid
    Invocation

    Called repeatedly, once for each input record.

    The function is called after the corresponding call(s) of transform() function.

    DescriptionReturns the component to the initial settings
    Example
    function void clean() {
       clear(customers);
    }
    integer countOnError(string errorMessage, string stackTrace)
    Requiredno
    Input Parametersstring errorMessage
    string stackTrace
    Returns

    The returned number defines the number of new output records that will be created by the transform() function.

    If the count() function returns 0, the subsequent call of transform() is skipped.

    InvocationCalled if count() throws an exception.
    Description

    For each input record it generates the number of output records that will be created from this input.

    If any of the input records causes fail of the count() function, and if user has defined another function (countOnError()), processing continues in this countOnError() at the place where count() failed.

    Example
    function integer countOnError(
                      string errorMessage, 
                      string stackTrace) {
       printErr(errorMessage);
       return 1;
    }
    integer transformOnError(string errorMessage, string stackTrace, integer idx)
    Requiredno
    Input Parametersstring errorMessage
    string stackTrace
    integer idx
    ReturnsInteger numbers. For more information, see Return Values of Transformations.
    InvocationCalled if transform() throws an exception.
    Description

    Creates output records.

    If any part of the transform() function for some output record causes fail of the transform() function, and if the user has defined another function (transformOnError()), processing continues in this transformOnError() at the place where transform() failed.

    The transformOnError() function gets the information gathered by transform() that was received from previously successfully processed code. Also the error message and stack trace are passed to transformOnError().

    Example
    function integer transformOnError(
                      string errorMessage, 
                      string stackTrace, 
                      integer idx) {
       printErr(errorMessage);
       printErr(stackTrace);	
       $out.0.OneCustomerOnError = customers[idx];
       $out.0.RecordNo = $recordNo;
       $out.0.OrderWithinRecord = idx;	
       return OK;
    }
    string getMessage()
    RequiredNo
    DescriptionPrints the error message specified and invoked by the user
    Invocation

    Called in any time specified by the user (called only when either count(), transform(), countOnError(), or transformOnError() returns value less than or equal to -2).

    Returnsstring
    void preExecute()
    RequiredNo
    Input parametersNone
    Returnsvoid
    Description

    May be used to allocate and initialize resources required by the transform. All resources allocated within this function should be released by the postExecute() function.

    InvocationCalled during each graph run before the transform is executed.
    void postExecute() 
    RequiredNo
    Input parametersNone
    Returnsvoid
    Description

    Should be used to free up any resources allocated within the preExecute() function.

    InvocationCalled during each graph run after the entire transform was executed.

    Access to input and output fields

    Input records or fields

    Input records or fields are accessible within the count(), countOnError(), transform() and transformOnError() functions only.

    Output records or fields

    Output records or fields are accessible within the transform() and transformOnError() functions only.

    [Warning]Warning

    All of the other CTL template functions allow to access neither inputs nor outputs.

    Remember that if you do not hold these rules, NPE will be thrown!

    Java Interface

    The transformation implements methods of the RecordNormalize interface and inherits other common methods from the Transform interface. See Common Java Interfaces and Public CloverDX API.

    Following are the methods of RecordNormalize interface:

    • boolean init(Properties parameters, DataRecordMetadata sourceMetadata, DataRecordMetadata targetMetadata)

      Initializes normalize class/function. This method is called only once at the beginning of normalization process. Any object allocation/initialization should happen here.

    • int count(DataRecord source)

      Returns the number of output records which will be created from specified input record.

    • int countOnError(Exception exception, DataRecord source)

      Called only if count(DataRecord) throws an exception.

    • int transform(DataRecord source, DataRecord target, int idx)

      idx is a sequential number of output record (starting from 0). For detailed information about return values and their meaning, see Return Values of Transformations. In Normalizer, only ALL, 0, SKIP, and Error codes have some meaning.

    • int transformOnError(Exception exception, DataRecord source, DataRecord target, int idx)

      Called only if transform(DataRecord, DataRecord, int) throws an exception.

    • void clean()

      Finalizes current round/clean after current round - called after the transform method was called for the input record.

    Examples

    Converting multivalue fields to multiple records

    Input records contain group name and list of users of the group. Convert records into tuples having group name and one username.

    accounting  | [johnsmith, elisabethtaylor]
    development | [georgegreen, janegreen, peterbrown]
    Solution

    Define the transformation using Normalize attribute.

    //#CTL2
    
    function integer count() {
        return length($in.0.users);
    }
    
    function integer transform(integer idx) {
        $out.0.group = $in.0.group;
        $out.0.user = $in.0.users[idx];
        return OK;
    }

    Normalizer will return following records:

    accounting |johnsmith
    accounting |elisabethtaylor
    development|georgegreen
    development|janegreen
    development|peterbrown

    Best Practices

    If the transformation is specified in an external file (with Normalize URL), we recommend users to explicitly specify Normalize source charset.