Version

    Incubation Functions

    Incubation category contains functions that have been recently developed and added. They are fully functional and supported, but there may be some backwards incompatible changes introduced in the future if needed, based on collected feedback. In time, these functions will leave incubation and will be moved in the standard function sections.

    In the functions that convert one data type to another, sometimes a format pattern of a date or any number must be defined. Also locale and time zone can have an influence on their formatting.

    • For detailed information about date formatting and/or parsing, see Date and Time Format.

    • For detailed information about formatting and/or parsing of any numeric data type, see Numeric Format.

    • For detailed information about locale, see Locale.

    • For detailed information about time zones, see Time Zone.

    [Note]Note

    Remember that numeric and date formats are displayed using system value Locale or Locale specified in the defaultProperties file, unless other Locale is explicitly specified. Similarly for Time zone.

    For more information on how Locale and Time zone may be changed in the defaultProperties, see Chapter 17, Engine Configuration.

    Here we provide the list of these functions:

    getAvroSchema

    Incubation

    string getAvroSchema(variant object);

    Converts variant data type to Avro schema. Resulted string contains JSON representation of Avro schema.

    This function should be used for one-time operations only. Using it together with parseAvro or writeAvro to generate schema for each processed record would reduce performance.

    Table 69.1. Schema generation from CTL types

    CTLAvroNote
    null valuenull typenull value is considered to be different type than any instance
    booleanboolean 
    byte, cbytebytes 
    datelong, logicalType: timestamp-millis 
    decimalbytes, logicalType: decimalprecision: 32, scale: 16
    integerint 
    longlong 
    numberdouble 
    stringstring 
    list[type1]array of type1items are of the same type type1
    list[type1, type2, ...]array of union of type1, type2, ...items are of different types (like null, string, map, ...)
    map{string -> type1}map(string) of type1keys are of type string; values are of the same type as type1
    map{string -> type1, type2, ...}recordkeys are of type string, keys become field names; values are of different types (like null, integer, string, list, map, ...)
    map{noString -> type1}map (string) of type1keys become strings; values are of the same type type1
    map{noString -> type1, type2, ...}map (string) of union of type1, type2, ...keys become strings; values are of different types (like null, string, map, ...)

    Compatibility

    VersionCompatibility Notice
    5.11.0

    getAvroSchema is available since 5.11.0 in incubation mode. It uses Apache Avro 1.10.1 Specification.

    See also:  parseAvro, writeAvro

    parseAvro

    Incubation

    variant parseAvro(byte avroData, string schema);

    Converts bytes containing Avro data serialized with the Binary encoding to a variant using the specified Avro schema in JSON. Avro data have to match Avro schema supplemented as the second parameter. The Avro data bytes are not regular Avro file, but the data only without Avro schema. The Avro data can be received for example from a messaging system (JMS) or events streaming system (Kafka).

    If the data input is null, the function returns null.

    If the Avro schema is null, the function fails with an error.

    Its counterpart is the function writeAvro.

    Table 69.2. Type conversion in parseAvro()

    Avro typeAvro logical typeResulted CTL typeNote
    null type null value 
    boolean boolean 
    int integer 
    intdatedatesystem timezone is used for conversion to Clover date
    inttime-millisdatesystem timezone is used for conversion to Clover date
    long long 
    longtime-microsdatesystem timezone is used for conversion to Clover date; micros are truncated
    longtimestamp-millisdate 
    longtimestamp-microsdatemicros are truncated
    longlocal-timestamp-millisdatesystem timezone is used for conversion to Clover date
    longlocal-timestamp-microsdatesystem timezone is used for conversion to Clover date; micros are truncated
    float number 
    double number 
    bytes byte 
    bytesdecimaldecimal 
    string string 
    stringuuidstring 
    record map {string -> any}keys match field names; apply this table to the value types
    enum string 
    array listapply this table to the items type
    map mapapply this table to the values type
    union types from unionapply this table to the union types
    fixed byte 
    fixeddecimaldecimal 
    fixedduration not supported

    Compatibility

    VersionCompatibility Notice
    5.11.0

    parseAvro is available since 5.11.0 in incubation mode. It uses Apache Avro 1.10.1 Specification.

    See also:  writeAvro, getAvroSchema

    writeAvro

    Incubation

    byte writeAvro(variant object, string schema);

    Converts variant data type to bytes contaning Avro data serialized with the Binary encoding using the specified Avro schema in JSON. Converted data have to match Avro schema supplemented as the second parameter. The resulted Avro data bytes are not regular Avro file, but the data only without Avro schema. The resulted Avro data can be used for example for a messaging system (JMS) or events streaming system (Kafka).

    If the data input is null, the function returns null.

    If the Avro schema is null, the function fails with an error.

    Its counterpart is the function parseAvro.

    Table 69.3. Type conversion in writeAvro()

    Avro typeAvro logical typeAccepted CTL typesNote
    null type null value 
    boolean boolean 
    int integer 
    intdateinteger, datesystem timezone is used for conversion Clover date to Avro date; integer value is written with no conversion
    inttime-millisinteger, datesystem timezone is used for conversion Clover date to Avro time; integer value is written with no conversion
    long long, integer 
    longtime-microslong, integer, datesystem timezone is used for conversion Clover date to Avro time; integer and long values are written with no conversion
    longtimestamp-millislong, integer, date 
    longtimestamp-microslong, integer, date 
    longlocal-timestamp-millislong, integer, datesystem timezone is used for conversion Clover date to Avro timestamp; integer and long values are written with no conversion
    longlocal-timestamp-microslong, integer, datesystem timezone is used for conversion Clover date to Avro timestamp; integer and long values are written with no conversion
    float number, decimal, long, integer 
    double number, decimal, long, integer 
    bytes byte, cbyte 
    bytesdecimalbyte, cbyte, decimal 
    string string 
    stringuuidstringvalue must be UUID
    record map {string -> any}keys match field names; apply this table to the value types
    enum stringvalue must match enum symbol
    array listapply this table to the items type
    map mapapply this table to the values type
    union types from unionapply this table to the union types
    fixed byte, cbyte 
    fixeddecimalbyte, cbyte, decimal 
    fixedduration not supported

    Compatibility

    VersionCompatibility Notice
    5.11.0

    writeAvro is available since 5.11.0 in incubation mode. It uses Apache Avro 1.10.1 Specification.

    Example 69.1. Usage of writeAvro

    byte avro;
    
    variant varMap = {"number" -> 1, "boolean" -> true, "string" -> "CloverDX"};
    string varSchema = getAvroSchema(varMap);
    byte varData = writeAvro(varMap, varSchema);
    variant varMapCopy = parseAvro(varData, varSchema);
    // varMap == varMapCopy
    
    integer varInteger = 1;
    //Avro schema for primitive type
    string varSchema = "\"int\"";
    byte varData = writeAvro(varInteger, varSchema);


    See also:  parseAvro, getAvroSchema