Version

    RESTConnector

    RESTClient 64x64

    Short description

    Ports

    Metadata

    RESTConnector attributes

    Details

    Best practices

    Compatibility

    See also

    Short description

    The RESTConnector component is designed to streamline the integration of REST APIs into CloverDX workflows. By leveraging OpenAPI specifications (OpenAPI documentation version 3.x.x in both JSON and YAML formats), this component pre-loads as many settings as possible directly from the API specification, reducing the amount of setup required. With its user-friendly interface and flexible configuration options, the RESTConnector enhances workflow efficiency and simplifies interactions with complex APIs, empowering users to work more effectively. For an example graph demonstrating the usage of the RESTConnector, please refer to our Tech Blog.

    The RESTConnector can be used without an OpenAPI specification, but this then requires manual configuration of all settings. This approach may reduce efficiency and hinder the overall user experience of the component.

    Same input metadata Sorted inputs Inputs Outputs Each to all outputs Java CTL Auto-propagated metadata

    -

    0-n

    0-n

    -

    x

    Ports

    Port type Number Required Description Metadata

    Input

    0-n

    Data to build request content and set component attributes

    Any

    Output

    0-n

    Parsed response content, status code, component attributes, …​

    Any

    Metadata

    RESTConnector does not propagate metadata.

    RESTConnector attributes

    Attribute Req Description Possible values

    Connection

    OpenAPI URL

    URL to the OpenAPI specification. The URL can point to a remote file (e.g., via HTTP or HTTPS protocols) as well as a local file (e.g., a file in sandbox). OpenAPI documentation version 3.x.x in both JSON and YAML formats is supported.

    API root URL

    Provide the root address of the target API - the address that will be called for each API invocation.

    Endpoint

    Select an endpoint from the pre-defined list.

    Request method

    Method of request. The method is populated based on the selected endpoint.

    GET (default) | POST | PUT | PATCH | DELETE | HEAD | OPTIONS | TRACE

    Request/response charset

    Character encoding of the input/output records. The default encoding depends on DEFAULT_CHARSET_DECODER in defaultProperties.

    UTF-8 | other encoding

    Authentication

    OAuth2 connection

    Create and authorize a new OAuth2 connection or link an existing authorized connection.

    Username (HTTP Basic)

    The username for accessing the REST API via HTTP Basic authentication. Only configure this when using HTTP Basic authentication.

    Password (HTTP Basic)

    The password to use when HTTP Basic authentication is needed. Only configure this when using HTTP Basic authentication.

    Bearer token

    You can specify a bearer token by:

    • Creating or linking a parameter: use the three dots button at the end of the field to access the Parameters editor. It is recommended to use a secure parameter for this purpose.

    • Creating a constant by manually typing it in.

    • Using the Input mapping to map it to the related attribute.

    API key value

    You can specify an API key value by:

    • Creating or linking a parameter: use the three dots button at the end of the field to access the Parameters editor. It is recommended to use a secure parameter for this purpose.

    • Creating a constant by manually typing it in.

    • Using the Input mapping to map it to the related attribute.

    API key location

    Select the location where the API key should be sent during each call: header or query.

    header | query

    API key parameter name

    Specify the name of the header or query parameter to be used for the API key.

    Request

    Request parameters

    Request parameters are automatically loaded when the selected endpoint provides them. You can assign parameter values in the Edit request parameters editor (accessed via the three dots) by creating a parameter reference or setting a static value. You can also map incoming records using Input mapping.

    JSON request body mapping

    When your endpoint requires a JSON object or array in the request body, you can map input records to the corresponding structure within the Mapping Editor (accessed via the three dots). The editor displays the expected JSON structure on the right side and the available input fields on the left side. You can drag and drop input fields onto the appropriate nodes within the JSON structure to define the mapping. Depending on the type of JSON request body (object or array), additional configuration might be necessary:

    • For JSON object bodies: When working with multiple input ports, you’ll typically also need to set the Max records per request value to '1'. This ensures that all input data is combined into a single JSON object, aligning with the expected structure.

    • For JSON array bodies: You can generally set the Max records per request to any value, as the mapping will handle the creation of an array of JSON objects based on the input data.

    Static request body

    To set a static request body, enter the full body contents in the format your API endpoint expects. You can reference parameters within the body to dynamically populate values.

    Max records per request

    The maximum number of records sent per one request. See JSON request body mapping above for more information.

    Multipart entities

    Fields that should be added as multipart entities to a POST request. The field name is used as the entity name. A semicolon-separated list of input fields is expected.

    Input mapping

    Use Input mapping to dynamically set request parameters or other attributes to values from incoming records, parameters, or dictionary entries. You can also use CTL functions to modify these values. Fields from input ports are only available when:

    Note that this mapping only allows you to access data from a single input port - the "driver" port that maps to the highest level of your request or port 0 if this cannot be determined by the component.

    HTTP headers

    HTTP header parameters are automatically loaded when the selected endpoint provides their specification. You can add any additional headers manually as needed.

    Input record sorting

    Used for JSON request body mapping to optimize efficiency and avoid caching by leveraging input record ordering. Configure the sort keys for each input port on separate tabs and ensure that the incoming data is pre-sorted based on these keys. For more details, see Sort key.

    Response

    JSON response mapping

    Allows you to visually configure parsing logic for the response provided by the API. The mapping allows you to parse even complex responses into multiple ports and also allows you to work with input ports and headers. The mapping can be defined for each HTTP status code or for groups of status code. This allows you to define separate mappings for different success responses as well as failure responses.

    This mapping can be used in conjunction with Default output mapping.

    The mapping lookup logic works as follows:

    • If the returned HTTP code matches an entry in the JSON response mapping, use the mapping to parse the response and route it appropriately.

    • If the returned HTTP code does not match an entry in the JSON response mapping:

      • If Default output mapping is defined, use it to process the response.

      • If Default output mapping is not defined:

        • If a HTTP 2xx response is returned, the call is treated as a success.

        • If a HTTP ≥300 response is returned, the component fails.

    Default output mapping

    A general-purpose mapping that is applied when no specific JSON response mapping is defined or when the response code doesn’t match any defined mapping. It can be used in conjunction with JSON response mapping to provide flexibility in handling various API responses.

    Pagination

    Allows you to specify and configure the pagination method. See Pagination for more information.

    No pagination (default), Offset and limit pagination, Page-based pagination, Token-based pagination

    Error handling

    Retry count

    How many times should the component retry a request in the case of a failure. The retry count specifies the number of times the component will attempt to re-send a failed request. A failure is considered to occur when the component encounters an error processing the request or response, such as an IOException. This does not include cases where the API returns an error status code (e.g., 500).

    0 (default)

    Retry delay

    Specifies the time intervals between retry attempts. It’s a comma-separated list of integers, where each integer represents the delay time in seconds for the corresponding retry. For example, a delay of 2, 5, 10 means the component will wait 2 seconds for the first retry, 5 seconds for the second, and 10 seconds for subsequent retries. If the number of retries exceeds the number of specified delays, the last delay in the list will be used for all subsequent retries.

    0 (default)

    Rate limit: time interval

    Limits the number of allowed requests defined by the Rate limit: max results property per time period (seconds, minutes, hours, days). When a limit is set and reached, the component waits until the defined time interval passes to continue sending requests.

    1s (default), 1m, 1h, 1d

    Rate limit: max requests

    Limits the total number of requests per time period defined by the Rate limit: time interval property. By default, the number of requests is unlimited. When a limit is set and reached, the component waits until the defined time interval passes to continue sending requests.

    Debug print

    When enabled, API calls and their corresponding responses will be logged to the execution log file. Use this feature with caution, as logging large amounts of data can significantly increase the size of the log file.

    false (default) | true

    Details

    Request parameters

    Request parameters can be mapped using a parameter reference or a constant value.

    RESTConnector request param mapping
    Figure 439. Request parameters example
    JSON request body mapping

    JSON request body mapping is used to map incoming records into body parameters. It must be populated when using multiple input ports.

    RESTConnector json request mapping
    Figure 440. JSON request body mapping example
    Static request body

    Example of a static request body - you can use parameter references instead of static values. Note that Static request body cannot be combined with JSON request body mapping.

    RESTConnector static body
    Figure 441. Static request body example
    Input mapping

    You can use Input mapping to map input records to request parameters or other attributes. Note that fields from input ports are only accessible in the editor under specific conditions:

    It’s also important to note that this mapping operates linearly, meaning it can only utilize fields from a single input port. This port is determined as follows: If a JSON request body mapping is defined at the highest level, the input port associated with that mapping will be used. Otherwise, the first mapped input port (port 0) will be utilized.

    RESTConnector input mapping
    Figure 442. Example input mapping
    JSON response output mapping

    This is the primary way of parsing data from API responses. This mapping allows you to visually map and process complex API responses based on the response schemas provided in the OpenAPI specification. The mapping allows you to parse each response type with its own mapping ensuring that the data you need is sent to correct output ports.

    The response types are distinguished based on their HTTP response code - i.e., you can map success response 200 in a different way than failure (HTTP 400) etc.

    To specify the mappings, start in the Select response codes to map panel in the upper-left corner of the Mapping tab. This panel allows you to add mappings for different response codes as defined in the OpenAPI specification for the API you are calling. Start with the Response type and work your way down to fill in additional details to create the mapping.

    • Response type is the type of the response that is associated with specific status code (e.g., 200 for success, 404 for "not found" etc.). This corresponds to part of the OpenAPI schema located at …​ / responses / <response type> / content / "application/json" / schema. The dropdown will list all response types provided by the API specification. Usually, you will see a success response here (e.g., "200" or even notation like "2XX") as well as error responses (e.g., "404", "4XX", etc.)

    • Response type filter allows you to specify a more detailed filter for the HTTP status codes. The mapping will then apply only to responses matching this filter. The filter can list specific codes one by one as a comma-separated list (e.g., "200,201,202") as well as ranges with a dash (e.g., "200-203"). These can be combined to allow you to easily list as many status codes in a single filter as needed.

      By default, the filter corresponds to the Response type (e.g. Response type "4xx" corresponds to filter "400-499"). You can modify the filter to exclude status codes or include additional ones as needed.

      This can be useful even in cases where the OpenAPI is not exact - for example, OpenAPI provides response code as 200 but the actual response is 201. In such case, you can change the filter to be "200,201".

      Similarly, for error handling it is common to use one schema for many errors, so you can write the filter as e.g., "400-499" to cover all statuses in the 4XX range.

    When the API response is parsed, the more specific codes are preferred instead of the ranges. For example, consider two mappings one of which is for "400-499", and the other one is for "404". This will effectively behave as if you wrote the mappings as "400-403,405-499" and "404". This approach allows you to easily cover many status codes while specifically cherry-picking the interesting ones that need to be handled differently.

    Each mapping can extract data from the response to multiple output ports. This works just like the mapping in the JSONExtract component. As an example, the following screenshot shows mapping of Petstore API and maps responses to four different ports. Ports are shown in the mapping tree just after the filter - so "HTTP_404 [1:1] (port 3)" means that response with status code 404 will be mapped to port 3 (and it is a 1:1 mapping of a single entity). The mapping maps the response like this:

    1. Valid pets (status codes 200 or 201) are mapped to port 0.

    2. Their photo URLs are sent to port 1.

    3. Their tags are sent to port 2.

    4. The 404 responses are mapped to port 3.

    RESTConnector json response mapping
    Figure 443. Example JSON response mapping

    In this example, status codes different from 200, 201, or 404 will not be matched and therefore the Default output mapping will be used instead.

    Default output mapping

    The Default output mapping is used whenever the response type (as defined by its status code) is not covered by any filters in the JSON response mapping. This allows you to map the complex and interesting responses in a simple, visual way via the JSON response mapping and handle all the other cases (even unknown responses and errors) with the Default output mapping.

    Default output mapping is specified as a transformation that allows you to use CTL and provides access to the response, its HTTP status code and even headers or component attributes. This gives you the ability to process all parts of the response as needed.

    The Default output mapping will also apply in cases where the API call failed on the network level or due to other runtime error - e.g., target server was not found, the call timed out and so on. In such cases, the errorMessage field will provide additional information about the failure. This is similar to the approach used in HTTPConnector’s output mapping.

    The following example sends all responses (not covered by the JSON response mapping, see its example) to port 4:

    RESTConnector default output mapping
    Figure 444. Example default output mapping

    If the Default output mapping is not defined and a response has status code ≥ 300 or if an error is encountered when calling the API, the component will fail and the error message will be written to the graph log.

    Pagination

    Consult your specific API documentation to determine the supported pagination type. Once you’ve identified the type, select the corresponding option from the list and populate the required configuration fields with the appropriate values from your API documentation.

    • Offset and limit pagination: uses a numerical offset to navigate through paginated results. The offset specifies the starting point for the next page of data.

    • Page-based pagination: uses a page number to navigate through paginated results. The page number identifies the specific page of data to be retrieved.

    • Token-based pagination: uses a unique token provided by the server to navigate through paginated results. The token identifies the next page of data.

    See below for an example configuration for CloverDX REST API, which uses the Offset and limit pagination. This configuration will return 25 results per page, starting from the very beginning (index 0), and it limits the total number of requests to 100.

    RESTConnector paging offset
    Figure 445. Offset and limit pagination configuration example

    Below is an example configuration for the X REST API, which uses the Token-based pagination. This configuration will return 25 results per page, and it does not limit the total number of requests.

    RESTConnector paging token
    Figure 446. Token-based pagination configuration example

    Best practices

    When configuring the RESTConnector, populate attributes from top to bottom. This sequential order ensures proper utilization of default values and streamlines the configuration process. By providing the OpenAPI URL first, the connector automatically loads available endpoints from the specification. Selecting an endpoint then triggers the automatic population of related parameters, data types, and other relevant information. This significantly simplifies subsequent steps like parameter mapping, request/response handling, and error handling, as many configuration options are pre-populated based on the selected endpoint. This approach promotes consistency, minimizes errors, and reduces debugging time, ultimately leading to more efficient and reliable integration flows.

    Compatibility

    Version Compatibility notice

    6.7.0

    RESTConnector is available since CloverDX version 6.7.