Version

    CloverDataReader

    Short Description
    Ports
    Metadata
    CloverDataReader Attributes
    Examples
    Compatibility
    See also

    Short Description

    CloverDataReader reads data stored in our internal binary CloverDX data format files. It can also read data from compressed files, or a dictionary.

    ComponentData sourceInput portsOutput portsEach to all outputsDifferent to different outputsTransformationTransf. req.JavaCTLAuto-propagated metadata
    CloverDataReaderCloverDX binary file11-n

    Ports

    Port typeNumberRequiredDescriptionMetadata
    Input0For Input Port Reading correct data recordsInclude specific byte/cbyte field
    Output0For correct data recordsAny
    1-nFor correct data recordsOutput 0

    Metadata

    CloverDataReader does not propagate metadata.

    CloverDataReader has no metadata template, but it can extract metadata from CloverDX file and propagate it forward as it would have a template. (Available since CloverETL 4.1.0-M1.)

    Metadata on the input port has to include a byte, cbyte or string field.

    Metadata on the output port has to be the same as metadata of data from the file.

    Metadata can use Autofilling Functions.

    CloverDataReader Attributes

    AttributeReqDescriptionPossible values
    Basic
    File URLyes

    An attribute specifying what data source(s) will be read (CloverDX data file, input port, dictionary). See Supported File URL Formats for Readers.

     
    Advanced
    Number of skipped records 

    Number of records to be skipped. See Selecting Input Records.

    0-N
    Max number of records 

    Maximum number of records to be read. See Selecting Input Records.

    0-N
    Number of skipped records per source Skip the first n records of each file.0-N
    Max number of records per source Reads maximally n records from each file.0-N
    Deprecated
    Index file URL 

    The name of an index file, including the path. If not specified, all records are read.

     
    Start record 

    Has exclusive meaning: Last record before the first that is already read. Has lower priority than Number of skipped records.

    0 (default) | 1-n
    Final record  Has inclusive meaning: Last record to be read. Has lower priority than Max number of records. all (default) | 1-n

    Examples

    Reading a CloverDX Data File
    Omitting Leading Records
    Omitting Leading Records of Each File
    Reading at most n Records in Total
    Reading at most n Records per File

    Reading a CloverDX Data File

    Read all records from the CloverDX data file.

    Solution

    Set up the File URL attribute.

    AttributeValue
    File URL${DATAIN_DIR}/my-clover-file.cdf

    CloverDataReader will read all the records from the file(s).

    Omitting Leading Records

    You have two CloverDX data files. First 3 records contain unimportant data and should not be read. The unimportant records are in the first file. (The records have been sorted and partitioned for example.)

    greengrocers1.cdf

    bread
    honey
    raisins
    pears
    plums

    greengrocers2.cdf

    carrot
    peas
    radish

    Solution

    Set up the File URL and Number of skipped records attributes.

    AttributeValue
    File URL${DATAIN_DIR}/greengrocers1.cdf;${DATAIN_DIR}/greengrocers2.cdf
    Number of skipped records3

    CloverDataReader reads the following items:

    pears
    plums
    carrot
    peas
    radish

    Omitting Leading Records of Each File

    There are two CloverDX data files: list1.cdf and list2.cdf. Each file starts with one record to be omitted.

    list1.cdf

    Goods
    cardigan
    shirt
    trousers

    list2.cdf

    Goods
    shoes
    sox

    Solution

    Set up the File URL and Number of skipped records per source attributes.

    AttributeValue
    File URL${DATAIN_DIR}/list1.cdf;${DATAIN_DIR}/list2.cdf
    Number of skipped records per source1

    CloverDataReader sends the following records to the output:

    cardigan
    shirt
    trousers
    shoes
    sox

    Reading at most n Records in Total

    You have three files stationery1.cdf, stationery2.cdf and stationery3.cdf and you need to read six records in total from all files.

    stationery1.cdf

    pen
    pencil
    marker
    paintbrush

    stationery2.cdf

    ink
    water colors
    oil colors

    stationery3.cdf

    notebook
    coloring book

    Solution

    Set up the File URL and Max number of records attributes.

    AttributeValue
    File URL${DATAIN_DIR}/stationery1.cdf;${DATAIN_DIR}/stationery2.cdf;${DATAIN_DIR}/stationery3.cdf
    Max number of records6

    CloverDataReader sends all 4 records from stationery1.cdf and 2 of 3 records from stationery2.cdf to the output port. No record from the file stationery3.cdf is sent to the output port as the limit has been reached already.

    pen
    pencil
    marker
    paintbrush
    ink
    water colors

    Reading at most n Records per File

    You have three CloverDX data files (stationery[1-3].cdf) from the previous example. Read at most 3 records from each file.

    Solution

    Set up the File URL and Max number of records per source attributes.

    AttributeValue
    File URL${DATAIN_DIR}/stationery1.cdf;${DATAIN_DIR}/stationery2.cdf;${DATAIN_DIR}/stationery3.cdf
    Max number of records per source3

    CloverDataReader reads 3 records from stationery1.cdf, 3 records from stationery2.cdf and 2 of 2 records from stationery3.cdf.

    pen
    pencil
    marker
    ink
    water colors
    oil colors
    notebook
    coloring book

    Compatibility

    VersionCompatibility Notice
    2.9

    CloverDataWriter also writes a header to output files with the version number. For this reason, CloverDataReader expects that files in CloverDX binary format contain such a header with the version number. CloverDataReader 2.9 cannot read files written by older versions nor these older versions can read data written by CloverDataWriter 2.9.

    4.0

    The internal structure of the zip archive has changed. Graphs that rely on the structure will stop working. Graphs that use plain zip file URL without internal entry specification are not affected: CloverDataReader with File URL zip:(${DATAOUT_DIR}/employees.zip)#DATA/employees will need to be fixed to use zip:(${DATAOUT_DIR}/employees.zip) instead.

    4.1.0-M1

    CloverDataReader can extract metadata template from CloverDX file. It can be seen as a metadata template corresponding to the file.

    4.4.0-M2

    CloverDataReader can read from input port just from byte or cbyte field.