Version

    71. Filtering the records

    Inserting the Filter
    Setting Up the Filter Component
    Two data streams

    In this chapter we will learn how to filter records with the Filter component.

    This chapter builds on the graph from the Creating a Transformation Graph

    Inserting the Filter

    The component for filtering is called Filter. You can find it in the Transformers category. Drag the component on the edge between FlatFileReader and SpredsheetWriter.

    filtering 0110
    Figure 510. Filter Component Attributes
    filtering 0120
    Figure 511. Filter was added to the graph

    The filtering condition is not specified yet, therefore can you see an error on the component.

    Setting Up the Filter Component

    Double-click the Filter component to open the component editor.

    filtering 0210
    Figure 512. Component Editor - Filter Component Attributes

    Click the Filter expression attribute row in the component editor. The Filter Editor will open:

    filtering 0220
    Figure 513. Filter Editor

    Select the salary item and double-click it. The $in.0.salary expression will appear on the blue background in the pane at the bottom.

    filtering 0230
    Figure 514. Selecting a Field

    Note that CloverDX validates the expressions automatically. Now an error will be found.

    Click on the right side from the $in.0.salary expression.

    Select a "greater" sign by clicking. It will appear in the pane on the right side from $in.0.salary.

    filtering 0250
    Figure 515. Selecting a "Greater" Sign

    You need to complete the expression to make it valid. Click at the right from the expression. Then type 24000. After typing the number, the expression becomes valid again.

    The number is the salary that will serve to filter incoming data flow. Only data records with salary higher than 24000 will be sent out.

    filtering 0260
    Figure 516. Filter Expression Defined

    By clicking OK, you close the FilterExpression editor.

    When you save the graph, you can see that the warning icon has disappeared from the Filter component:

    filtering 0270
    Figure 517. Warning Icon Has Disappeared

    The Filter is configured and you can run the graph.

    Best practices

    It’s better to filter and sort records than to sort and filter.

    If you need to split data into multiple (more than two) streams, use Partitioner.

    See also

    Documentation on ExtFilter

    Two data streams

    You can use Filter component to split data stream into two data streams. Connect an edge to the second output port of the filter to get rejected records too.

    filtering 0310
    Figure 518. Splitting one data stream

    You can use the same condition as in the previous example. The records matching the data filter condition will be passed to the first output port, the later ones will go to the second output port.