CloverDX Designer > CTL2 - CloverDX Transformation Language > Functions Reference > Incubation Functions

Incubation Functions

List of functions

getAvroSchema	writeAvro
parseAvro

Incubation category contains functions that have been recently developed and added. They are fully functional and supported, but there may be some backwards incompatible changes introduced in the future if needed, based on collected feedback. In time, these functions will leave incubation and will be moved in the standard function sections.

In the functions that convert one data type to another, sometimes a format pattern of a date or any number must be defined. Also locale and time zone can have an influence on their formatting.

For detailed information about date formatting and/or parsing, see Date and Time Format.
For detailed information about formatting and/or parsing of any numeric data type, see Numeric Format.
For detailed information about locale, see Locale.
For detailed information about time zones, see Time Zone.

Note

	Note
Remember that numeric and date formats are displayed using system value Locale or Locale specified in the `defaultProperties` file, unless other Locale is explicitly specified. Similarly for Time zone. For more information on how Locale and Time zone may be changed in the `defaultProperties`, see Chapter 17, Engine Configuration.

Remember that numeric and date formats are displayed using system value Locale or Locale specified in the defaultProperties file, unless other Locale is explicitly specified. Similarly for Time zone.

For more information on how Locale and Time zone may be changed in the defaultProperties, see Chapter 17, Engine Configuration.

Here we provide the list of these functions:

getAvroSchema

Incubation

string getAvroSchema(variant object);

Converts variant data type to Avro schema. Resulted string contains JSON representation of Avro schema.

This function should be used for one-time operations only. Using it together with parseAvro or writeAvro to generate schema for each processed record would reduce performance.

Table 69.1. Schema generation from CTL types

CTL	Avro	Note
null value	null type	null value is considered to be different type than any instance
boolean	boolean
byte, cbyte	bytes
date	long, logicalType: timestamp-millis
decimal	bytes, logicalType: decimal	precision: 32, scale: 16
integer	int
long	long
number	double
string	string
list[type1]	array of type1	items are of the same type type1
list[type1, type2, ...]	array of union of type1, type2, ...	items are of different types (like null, string, map, ...)
map{string -> type1}	map(string) of type1	keys are of type string; values are of the same type as type1
map{string -> type1, type2, ...}	record	keys are of type string, keys become field names; values are of different types (like null, integer, string, list, map, ...)
map{noString -> type1}	map (string) of type1	keys become strings; values are of the same type type1
map{noString -> type1, type2, ...}	map (string) of union of type1, type2, ...	keys become strings; values are of different types (like null, string, map, ...)

Compatibility

Version	Compatibility Notice
5.11.0	getAvroSchema is available since 5.11.0 in incubation mode. It uses Apache Avro 1.10.1 Specification.

See also: parseAvro, writeAvro

parseAvro

Incubation

variant parseAvro(byte avroData, string schema);

Converts bytes containing Avro data serialized with the Binary encoding to a variant using the specified Avro schema in JSON. Avro data have to match Avro schema supplemented as the second parameter. The Avro data bytes are not regular Avro file, but the data only without Avro schema. The Avro data can be received for example from a messaging system (JMS) or events streaming system (Kafka).

If the data input is null, the function returns null.

If the Avro schema is null, the function fails with an error.

Its counterpart is the function writeAvro.

Table 69.2. Type conversion in parseAvro()

Avro type	Avro logical type	Resulted CTL type	Note
null type		null value
boolean		boolean
int		integer
int	date	date	system timezone is used for conversion to Clover date
int	time-millis	date	system timezone is used for conversion to Clover date
long		long
long	time-micros	date	system timezone is used for conversion to Clover date; micros are truncated
long	timestamp-millis	date
long	timestamp-micros	date	micros are truncated
long	local-timestamp-millis	date	system timezone is used for conversion to Clover date
long	local-timestamp-micros	date	system timezone is used for conversion to Clover date; micros are truncated
float		number
double		number
bytes		byte
bytes	decimal	decimal
string		string
string	uuid	string
record		map {string -> any}	keys match field names; apply this table to the value types
enum		string
array		list	apply this table to the items type
map		map	apply this table to the values type
union		types from union	apply this table to the union types
fixed		byte
fixed	decimal	decimal
fixed	duration		not supported

Compatibility

Version	Compatibility Notice
5.11.0	parseAvro is available since 5.11.0 in incubation mode. It uses Apache Avro 1.10.1 Specification.

See also: writeAvro, getAvroSchema

writeAvro

Incubation

byte writeAvro(variant object, string schema);

Converts variant data type to bytes contaning Avro data serialized with the Binary encoding using the specified Avro schema in JSON. Converted data have to match Avro schema supplemented as the second parameter. The resulted Avro data bytes are not regular Avro file, but the data only without Avro schema. The resulted Avro data can be used for example for a messaging system (JMS) or events streaming system (Kafka).

If the data input is null, the function returns null.

If the Avro schema is null, the function fails with an error.

Its counterpart is the function parseAvro.

Table 69.3. Type conversion in writeAvro()

Avro type	Avro logical type	Accepted CTL types	Note
null type		null value
boolean		boolean
int		integer
int	date	integer, date	system timezone is used for conversion Clover date to Avro date; integer value is written with no conversion
int	time-millis	integer, date	system timezone is used for conversion Clover date to Avro time; integer value is written with no conversion
long		long, integer
long	time-micros	long, integer, date	system timezone is used for conversion Clover date to Avro time; integer and long values are written with no conversion
long	timestamp-millis	long, integer, date
long	timestamp-micros	long, integer, date
long	local-timestamp-millis	long, integer, date	system timezone is used for conversion Clover date to Avro timestamp; integer and long values are written with no conversion
long	local-timestamp-micros	long, integer, date	system timezone is used for conversion Clover date to Avro timestamp; integer and long values are written with no conversion
float		number, decimal, long, integer
double		number, decimal, long, integer
bytes		byte, cbyte
bytes	decimal	byte, cbyte, decimal
string		string
string	uuid	string	value must be UUID
record		map {string -> any}	keys match field names; apply this table to the value types
enum		string	value must match enum symbol
array		list	apply this table to the items type
map		map	apply this table to the values type
union		types from union	apply this table to the union types
fixed		byte, cbyte
fixed	decimal	byte, cbyte, decimal
fixed	duration		not supported

Compatibility

Version	Compatibility Notice
5.11.0	writeAvro is available since 5.11.0 in incubation mode. It uses Apache Avro 1.10.1 Specification.

Example 69.1. Usage of writeAvro

byte avro;

variant varMap = {"number" -> 1, "boolean" -> true, "string" -> "CloverDX"};
string varSchema = getAvroSchema(varMap);
byte varData = writeAvro(varMap, varSchema);
variant varMapCopy = parseAvro(varData, varSchema);
// varMap == varMapCopy

integer varInteger = 1;
//Avro schema for primitive type
string varSchema = "\"int\"";
byte varData = writeAvro(varInteger, varSchema);

See also: parseAvro, getAvroSchema