Multivalue Fields
Lists and Maps Support in Components |
Joining on Lists and Maps (Comparison Rules) |
Each metadata field commonly stores only one value, e.g. one integer, one string, one date, etc. However, you can also set one field to carry more values of the same type.
Note | |
---|---|
Multivalue fields is a new feature available as of version 3.3. |
Example 31.3. Example situations when you could take advantage of multivalue fields
A record containing an employee's
ID
,Name
andAddress
. Since employees move from time to time, you might need to keep track of all their addresses, both current and past. Instead of creating new metadata fields each time an employee moves to a new address, you can store a list of all addresses into one field.You are processing an input stream of
CSV
files, each containing a different column count. Normally, that would imply creating new metadata for each file (each column count). Instead, you can define a generic map in metadata and append fields to it each time they occur.
As implied above, there are two types of structures:
list - is a set containing elements of a given data type (any you want). In source code, lists are marked by the [] brackets, for example:
integer[] list1 = [1, 367, -1, 20, 5, 0, -79]; // a list of integer elements boolean[] list2 = [true, false, randomBoolean()]; // a list of three boolean elements string[] list3; // a just-declared empty list to be filled by strings
map - is a pair of keys and their values.
A key is always a string
while a value can be any data type -
but you cannot mix them (remember a map holds values of the same type).
Example:
map[string,date] dateMap; // declaration // filling the map with values dateMap["a"] = 2011-01-01; dateMap["b"] = 2012-12-31; dateMap["c"] = randomDate(2011-01-01,2012-12-31);
For more information about maps and lists, see Data Types in CTL2.
Important | |
---|---|
To change a field from single-value to multi-value:
|
Lists and Maps Support in Components
An alphabetically sorted list of components which you can use multivalue fields in:
Component | List | Map | Note |
---|---|---|---|
CloverDataReader | ✓ | ✓ | |
CloverDataWriter | ✓ | ✓ | |
ParallelSimpleGather | ✓ | ✓ | Round robin |
✓ | ⨯ | Merge by key | |
✓ | ✓ | Simple gather | |
ParallelPartition | ⨯ | ⨯ | Ranges |
✓ | ⨯ | Partition key | |
✓ | ✓ | Partition class | |
Concatenate | ✓ | ✓ | |
DataGenerator | ✓ | ✓ | |
DataIntersection | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
DBJoin | ✓ | ✓ | Map is not a part of key |
Dedup | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
Denormalizer | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
Filter | ✓ | ✓ | |
ExtHashJoin | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
ExtMergeJoin | ✓ | ✓ | Map is not a part of key |
ExtSort | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key. | |
JavaBeanReader | ✓ | ⨯ | |
JavaBeanWriter | ✓ | ⨯ | |
JavaMapWriter | ✓ | ✓ | |
JSONExtract | ✓ | ⨯ | |
JSONReader | ✓ | ⨯ | |
JSONWriter | ✓ | ⨯ | |
LookupJoin | ✓ | ✓ | |
LookupTableReaderWriter | ✓ | ✓ | |
Merge | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
Normalizer | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
Partition | ⨯ | ⨯ | Ranges |
✓ | ⨯ | Partition key | |
✓ | ✓ | Partition class | |
Reformat | ✓ | ✓ | |
RelationalJoin | ✓ | ✓ | Map is not a part of key |
Rollup | ✓ | ⨯ | Sorted input |
✓ | ✓ | Sorted input, map not part of key | |
✓ | ✓ | Unsorted input | |
SequenceChecker | ✓ | ⨯ | |
✓ | ✓ | Map is not a part of key | |
SimpleCopy | ✓ | ✓ | |
SimpleGather | ✓ | ✓ | |
Sleep | ✓ | ✓ | |
SortWithinGroups | ✓ | ⨯ | |
XMLExtract | ✓ | ⨯ | |
XMLReader | ✓ | ⨯ | |
XMLWriter | ✓ | ✓ |
At the moment, neither map
nor list
structures
can be extracted as metadata from flat files.
Joining on Lists and Maps (Comparison Rules)
You can specify fields that are lists or maps as Join keys (see Join Types) just like any other fields. The only question is when two maps (lists) equal.
A list/map can:
be
null
- it is not specified;map[string,date] myMap; // a just-declared map - no keys, no values
contain empty elements
string[] myList = ["hello", ""]; // a list whose second element is empty
contain n elements - an ordinary case described, e.g. in Example 31.3, “Example situations when you could take advantage of multivalue fields”
Two maps (lists) are equal if
both of them are not null
, they have the same data type, element count
and all element values (keys-values in maps) are equal.
Two maps (lists) are not equal if
either of them is null
.
Important | |
---|---|
When comparing two lists, the order of their elements has to match, too. In maps, there is no 'order' of elements, therefore you cannot use them in Sort key. |
Example 31.4. Integer lists which are (not) equal - symbolic notation
[1,2] == [1,2] [null] != [1,2] [1] != [1,2] null != null // two unspecified lists [null] == [null] // an extra case: lists which are not empty but whose elements are null
Note: Maps are implemented as LinkedHashMap
,
thus their properties derive from it.