Incubation

KafkaWriter writes events (messages) to a Kafka cluster.

If an error port is not connected and an error occurs, the component fails.

KafkaWriter propagates metadata from left to right, i.e. from input port to output port.

KafkaWriter has a metadata template on its input and error ports.

The input port metadata template contains fields required for creation of a Kafka record (topic, partition, key, value and timestamp).

The error port metadata template contains fields required for creation of a Kafka record (topic, partition, key, value and timestamp) plus additional fields offset, errorMessage and stacktrace.

You can use the fields partition, offset and timestamp from SuccessMetadata in the Output mapping to get the actual values of the written record. These fields will often not be present in the input data.

The ErrorMetadata are the same as SuccessMetadata with additional fields for troubleshooting of failed records - the error message and stacktrace.

KafkaWriter allows you to produce (write) events to Kafka using the Kafka Producer API.

The events coming from the input port are written to a Kafka topic specified either via the Topic attribute or in the Input mapping.

Topic partition is assigned automatically (by default), or can be also overridden via the Partition attribute or in the Input mapping.

If an output port is connected, records successfully written to Kafka are also written to the connected edge. Similarly, failed records are written to the error port edge, if connected.

Besides the configuration available through the component attributes, any other Kafka producer property can be configured in the Producer configuration attribute.