Version

    Kafka Message Listeners

    Overview

    Kafka Message Listeners allow you to listen for incoming events in specified Kafka topic(s) and trigger tasks in reaction to these incoming Kafka events.

    This can be used namely to automatically process these Kafka events using a job as a triggered task.

    The listener offers two ways of specifying which Kafka connection to use. Either by using an existing connection configuration file (.cfg), or by using an existing job file (graph, jobflow) with a Kafka connection defined.

    In the first case, an additional parameters file can be selected to resolve any potential parameters used in the connection.

    Users have to ensure no other consumer is subscribed to the same configuration as the listener. Or else the listener may fail.

    Kafka message listener subscribes to a topic and waits for appearance of new events with a timeout defined by configuration property clover.event.kafkaPollInterval. The poll either ends when new events appear - the Kafka connection is closed and the specified task is triggered, or after this interval passes - the Kafka connection is re-created.

    This interval can be raised or lowered if the default value is undesirable.

    Kafka Message Listener Attributes

    Below is the list of attributes which can be set up for Kafka Message Listeners:

    Table 56. Attributes of Kafka Message Listener
    Attribute Description

    Name

    The listener name.

    Owner

    Clover user to use for the listener.

    Enabled

    The state of listener.

    Kafka Connection

    Load connection from

    Possible values are:

    • configuration file (.cfg) - use an externalized Kafka connection from a .cfg file.

    • job file (.grf, .jbf) - use a Kafka connection from a job file.

    Sandbox

    The sandbox which contains the Kafka configuration file or job file.

    Connection file

    Connection configuration file (.cfg) which defines the Kafka cluster connection.

    Available only if configuration file (.cfg) is selected.

    Parameters file

    An additional parameters file used to resolve any potential parameters used in the connection.

    The default workspace.prm parameter file in the selected sandbox is always used for parameter resolution. Using this option, you can also select an additional parameter file.

    Optional, and available only if configuration file (.cfg) is selected.

    Job file

    Job file (graph, jobflow) which contains a Kafka cluster connection.

    Available only if job file (.grf, .jbf) is selected.

    Connection

    A Kafka connection to use.

    Available only if job file (.grf, .jbf) is selected.

    Kafka Topic(s)

    Topic specified by

    Possible values are:

    • name - select topic by a single topic name or a semicolon-separated list of topic names.

    • regular expression - select topic(s) by a pattern used to match topic names.

    Topic name(s)

    Kafka topic, which will be used to listen for events. Possible values are: a single topic name or a semicolon-separated list of topic names.

    Available only if Topic specified by - name is selected.

    Topic pattern

    Kafka topic name pattern (regular expression). Matched topics will be used to listen for events.

    Available only if Topic specified by - regular expression is selected.

    Group name

    A Kafka consumer group to use for the listener.

    Parameters passed from the listener to the task

    The listener passes a set of predefined parameters to the triggered task. Usage of these parameters allows you to write more generic jobs to process the Kafka events.

    For example, you can use parameter EVENT_KAFKA_TOPICS in Kafka components to work with the same topics as the listener.

    Table 57. Parameters passed from the listener to the task
    Parameter Description

    EVENT_USERNAME

    The name of the user who caused the event.

    EVENT_USER_ID

    A numeric ID of the user who caused the event.

    EVENT_KAFKA_LISTENER_ID

    An ID of the listener which triggered the event.

    EVENT_KAFKA_TOPICS

    Kafka topic name(s) defined in the listener which triggered the event.

    EVENT_KAFKA_TOPICS_PATTERN

    Kafka topic name pattern defined in the listener which triggered the event.

    EVENT_KAFKA_GROUP

    Kafka consumer group defined in the listener which triggered the event.

    Permissions

    To be able to use Kafka message listeners, the user need to have the proper group permissions assigned. These permissions are:

    • List Kafka message listeners limited/unlimited - for listing and viewing existing listeners, either those belonging to a user-owned sandbox, or all listeners.

    • Create Kafka message listener - for creating new listeners.

    • Edit Kafka message listener - for editing existing listeners.

    • Delete Kafka message listener - for deleting existing listeners.