Version

    Multivalue Fields

    Lists and Maps Support in Components
    Joining on Lists and Maps (Comparison Rules)

    Each metadata field commonly stores only one value, e.g. one integer, one string, one date, etc. However, you can also set one field to carry more values of the same type.

    [Note]Note

    Multivalue fields is a new feature available as of version 3.3.

    Example 32.4. Example situations when you could take advantage of multivalue fields

    • A record containing an employee's ID, Name and Address. Since employees move from time to time, you might need to keep track of all their addresses, both current and past. Instead of creating new metadata fields each time an employee moves to a new house, you can store a list of all addresses into one field.

    • You are processing an input stream of CSV files, each containing a different column count. Normally, that would imply creating new metadata for each file (each column count). Instead, you can define a generic map in metadata and append fields to it each time they occur.


    As implied above, there are two types of structures:

    list - is a set containing elements of a given data type (any you want). In source code, lists are marked by the [] brackets. Example:

    integer[] list1 = [1, 367, -1, 20, 5, 0, -79]; // a list of integer elements 
    boolean[] list2 = [true, false, randomBoolean()]; // a list of three boolean elements
    string[] list3; // a just-declared empty list to be filled by strings

    map - is a pair of keys and their values. A key is always a string while a value can be any data type - but you cannot mix them (remember a map holds values of the same type). Example:

    map[string,date] dateMap; // declaration
    
    // filling the map with values
    dateMap["a"] = 2011-01-01;
    dateMap["b"] = 2012-12-31;
    dateMap["c"] = randomDate(2011-01-01,2012-12-31);

    For more information about maps and lists, see Data Types in CTL2.

    [Important]Important

    To change a field from single-value to multi-value:

    1. Go to Metadata Editor.

    2. Click a field or create a new one.

    3. In PropertyBasic, switch Container Type either to list or map. (You will see an icon appears next to the field Type in the left hand record pane.)

    Lists and Maps Support in Components

    An alphabetically sorted list of components which you can use multivalue fields in:

    ComponentListMapNote
    CloverDataReader
    yes
    yes
     
    CloverDataWriter
    yes
    yes
     
    ParallelSimpleGather

    yes

    yes

    Round robin

    yes

    no

    Merge by key

    yes

    yes

    Simple gather
    ParallelPartition

    no

    no

    Ranges

    yes

    no

    Partition key

    yes

    yes

    Partition class
    Concatenate
    yes
    yes
     
    DataGenerator
    yes
    yes
     
    DataIntersection

    yes

    no

     

    yes

    yes

    Map is not a part of key
    DBJoin
    yes
    yes
    Map is not a part of key
    Dedup

    yes

    no

     

    yes

    yes

    Map is not a part of key
    Denormalizer

    yes

    no

     

    yes

    yes

    Map is not a part of key
    Filter
    yes
    yes
     
    ExtHashJoin

    yes

    no

     

    yes

    yes

    Map is not a part of key
    ExtMergeJoin
    yes
    yes
    Map is not a part of key
    ExtSort

    yes

    no

     

    yes

    yes

    Map is not a part of key.
    JavaBeanReader
    yes
    no 
    JavaBeanWriter
    yes
    no 
    JavaMapWriter
    yes
    yes
     
    JSONExtract
    yes
    no 
    JSONReader
    yes
    no 
    JSONWriter
    yes
    no 
    LookupJoin
    yes
    yes
     
    LookupTableReaderWriter
    yes
    yes
     
    Merge

    yes

    no

     

    yes

    yes

    Map is not a part of key
    Normalizer

    yes

    no

     

    yes

    yes

    Map is not a part of key
    Partition

    no

    no

    Ranges

    yes

    no

    Partition key

    yes

    yes

    Partition class
    Reformat
    yes
    yes
     
    RelationalJoin
    yes
    yes
    Map is not a part of key
    Rollup

    yes

    no

    Sorted input

    yes

    yes

    Sorted input, map not part of key

    yes

    yes

    Unsorted input
    SequenceChecker

    yes

    no

     

    yes

    yes

    Map is not a part of key
    SimpleCopy
    yes
    yes
     
    SimpleGather
    yes
    yes
     
    Sleep
    yes
    yes
     
    SortWithinGroups
    yes
    no 
    XMLExtract
    yes
    no 
    XMLReader
    yes
    no 
    XMLWriter
    yes
    yes
     

    At the moment, neither map nor list structures can be extracted as metadata from flat files.

    Joining on Lists and Maps (Comparison Rules)

    You can specify fields that are lists or maps as Join keys (see Join Types) just like any other fields. The only question is when two maps (lists) equal.

    First of all, let us clarify this. A list/map can:

    Two maps (lists) are equal if both of them are not null, they have the same data type, element count and all element values (keys-values in maps) are equal.

    Two maps (lists) are not equal if either of them is null.

    [Important]Important

    When comparing two lists, the order of their elements has to match, too. In maps, there is no 'order' of elements and therefore you cannot use them in Sort key.

    Example 32.5.  Integer lists which are (not) equal - symbolic notation

    [1,2] == [1,2]
    [null] != [1,2]
    [1] != [1,2]
    null != null // two unspecified lists
    [null] == [null] // an extra case: lists which are not empty but whose elements are null


    Note: Maps are implemented as LinkedHashMap and thus their properties derive from it.