Version

    Chapter 37. Simple HTTP API

    The Simple HTTP API is a basic Server automation tool that lets you control the Server from external applications using simple HTTP calls.

    Most operations are accessible using the HTTP GET method and return plain text. Thus, both request and response can be conveniently sent and parsed using very simple tools (wget, grep, etc.).

    If global security is on (on by default), the Basic HTTP authentication is used. Authenticated operations will require valid user credentials with corresponding permissions.

    Note that the Graph-related operations graph_run, graph_status and graph_kill also work for jobflows and Data Profiler jobs.

    The generic pattern for a request URL: 

    http://[domain]:[port]/[context]/[servlet]/[operation]?[param1]=[value1]&[param2]=[value2]...

    example: http://localhost:8080/clover/simpleHttpApi/help

    [Note]Note

    For backward compatibility, you can also use http://localhost:8080/clover/request_processor/help.

    CSRF Protection

    The Simple HTTP API provides protection against Cross-Site Request Forgery (CSRF) attacks. An example of such an attack is a case where the user is logged into the Server Console, and an attacker sends him a link to the Simple HTTP API such that it runs a graph. Clicking on such a link would call the Simple HTTP API and re-use the session of the logged-in user. There are also more complex variants of the attack that are harder to detect by the user.

    The protection against such an attack is that the Simple HTTP API requires the presence of the X-Requested-By header in the HTTP request. Value of the header can be arbitrary (it is not checked). Such a header cannot be set by CSRF attack vectors, i.e. by clicking on a link in an email.

    Examples of calling the API with the X-Requested-By header: 

    curl --header "X-Requested-By: arbitrary_value"  http://user:password@hostname:port/clover/simpleHttpApi/graph_run?sandbox=project&graphID=migration.grf

    wget --header "X-Requested-By: arbitrary_value" --user=$USER --password=$PASS -O ./$OUTPUT_FILE $REQUEST_URL

    The CSRF protection of Simple HTTP API can be disabled via the security.csrf.protection.enabled configuration property. It is enabled by default. If the protection is disabled, it is not necessary to set the X-Requested-By header.

    The Server Console's page for testing the Simple HTTP API uses a different CSRF protection mechanism. The requests contain a csrftoken parameter. This is intended for usage only in the testing page.

    List of Operations

    The HTTP API is enabled by default. You can disable it with the configuration property http.api.enabled. In the Server GUI, switch to Configuration → Setup and add the following line 

    http.api.enabled=false

    to the properties file.

    Operation help

    parameters

    no

    returns

    a list of possible operations and parameters with its descriptions

    example

    http://localhost:8080/clover/simpleHttpApi/help

    Operation graph_run

    Call this operation to start an execution of the specified job. The operation is called graph_run for backward compatibility, however it may execute a graph, jobflow or profiler job.

    parameters

    Table 37.1. Parameters of graph_run

    NameMandatoryDefaultDescription
    graphIDyes-The file path to the job file, relative to the sandbox root.
    sandboxyes-The text ID of sandbox.
    additional job parametersno 

    Any URL parameter with the param_ prefix is passed to the executed job and may be used in transformation XML as a placeholder, but without the param_ prefix. e.g. param_FILE_NAME specified in URL may be used in the XML as ${FILE_NAME}. These parameters are resolved only during loading of XML, so it cannot be pooled.

    additional config parametersno 

    URL parameters prefixed with config_ can set some of the execution parameters. For graphs, the following parameters are supported:

    • config_skipCheckConfig - when set to false, graph configuration will be checked before the execution.

    • config_logLevel - log level of the executed graph, one of OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL.

    • config_clearObsoleteTempFiles - when set to true, temp files of previous runs of this graph will be deleted before the execution.

    • config_debugMode - when set to true, debug mode for a given graph will be enabled. For more information, see Job Config Properties.

    nodeIDno-

    In Cluster mode, it is the ID of a node which should execute the job. However it is not final. If the graph is distributed or the node is disconnected, the graph may be executed on another node.

    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    run ID: incremental number, which identifies each execution request

    example

    http://localhost:8080/clover/simpleHttpApi/graph_run?graphID=graph/graphDBExecute.grf&sandbox=mva

    Operation graph_status

    Call this operation to obtain a status of a specified job execution. The operation is called graph_status for backward compatibility; however, it may return status of a graph or jobflow.

    parameters

    Table 37.2. Parameters of graph_status

    NameMandatoryDefaultDescription
    runIDyes-Id of each graph execution
    returnTypenoSTATUSSTATUS | STATUS_TEXT | DESCRIPTION | DESCRIPTION_XML
    waitForStatusno-

    Status code which we want to wait for. If it is specified, this operation will wait until the graph is in the required status.

    waitTimeoutno0

    If waitForStatus is specified, it will wait only for the specified amount of milliseconds. Default 0 means forever, but it depends on an application server configuration. When the specified timeout expires and graph run still isn't in a required status, the server returns code 408 (Request Timeout). 408 code may be also returned by an application server if its HTTP request timeout expires before.

    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    Status of a specified graph. It may be a number code, text code or a complex description in dependence on the optional parameter returnType. Description is returned as a plain text with a pipe as a separator, or as XML. A schema describing XML format of the XML response is accessible on CloverDX Server URL: http://[host]:[port]/clover/schemas/executions.xsd Depending on the waitForStatus parameter, it may return a result immediately or wait for a specified status.

    example 

    http://localhost:8080/clover/simpleHttpApi/graph_status?runID=123456&returnType=DESCRIPTION&waitForStatus=FINISHED&waitTimeout=60000

    Operation graph_kill

    Call this operation to abort/kill a job execution. The operation is called graph_kill for backward compatibility; however, it may abort/kill a graph, jobflow or profiler job.

    parameters

    Table 37.3. Parameters of graph_kill

    NameMandatoryDefaultDescription
    runIDyes-The ID of each graph execution
    returnTypenoSTATUSSTATUS | STATUS_TEXT | DESCRIPTION
    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    The status of the specified graph after an attempt to kill it. It may be a number code, text code or a complex description in dependence on optional parameter.

    example

    http://localhost:8080/clover/simpleHttpApi/graph_kill?runID=123456&returnType=DESCRIPTION

    Operation server_jobs

    parameters

    no

    returns

    a list of runIDs of currently running jobs.

    example

    http://localhost:8080/clover/simpleHttpApi/server_jobs

    Operation sandbox_list

    parameters

    no

    returns

    List of all sandbox text IDs. In the next versions, it will return only accessible ones.

    example

    http://localhost:8080/clover/simpleHttpApi/sandbox_list

    Operation sandbox_content

    parameters

    Table 37.4. Parameters of sandbox_content

    NameMandatoryDefaultDescription
    sandboxyes-text ID of sandbox
    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    A list of all elements in the specified sandbox. Each element may be specified as a file path relative to the sandbox root.

    example

    http://localhost:8080/clover/simpleHttpApi/sandbox_content?sandbox=mva

    Operation executions_history

    parameters

    Table 37.5. Parameters of executions_history

    NameMandatoryDefaultDescription
    fromno-

    Lower datetime limit of start of execution. The operation will return only records after (and equal to) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).

    tono-

    The upper datetime limit of start of execution. The operation will return only records before (and equal to) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).

    stopFromno-

    The lower datetime limit of stop of execution. The operation will return only records after (and equal to) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).

    stopTono-

    The upper datetime limit of stop of execution. The operation will return only records before (and equal to) this datetime. Format: "yyyy-MM-dd HH:mm" (must be URL encoded).

    statusno-

    Current execution status. The operation will return only records with specified STATUS. The values are RUNNING | ABORTED | FINISHED_OK | ERROR

    sandboxno-

    Sandbox code. The operation will return only records for graphs from a specified sandbox.

    jobFileno-

    The text Id, which is unique in a specified sandbox. The file path is relative to the sandbox root.

    orderByno-

    An attribute for list ordering. Possible values: id | graphId | status | startTime | stopTime. By default, there is no ordering.

    orderDescendnotrue

    A switch which specifies ascending or descending ordering. If true (default), ordering is descending.

    returnTypenoIDs

    Possible values are: IDs | DESCRIPTION | DESCRIPTION_XML | COUNT

    indexno0An index of the first returned records in a whole record set.
    recordsno1000

    The maximum amount of returned records. Set the value to unlimited for parameter to return an unlimited number of records.

    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    List of executions according to filter criteria.

    For returnType==IDs returns a simple list of runIDs (with new line delimiter).

    For returnType==COUNT returns only the number of entries. Note that filtering works, but the parameters orderBy, orderDescend, index and records are silently ignored.

    For returnType==DESCRIPTION returns complex response which describes current status of selected executions, their phases, nodes and ports. 

    execution|[runID]|[status]|[username]|[sandbox]|[graphID]|[startedDatetime]|[finishedDatetime]|[clusterNode]|[graphVersion](1)
phase|[index]|[execTimeInMilis]
node|[nodeID]|[status]|[totalCpuTime]|[totalUserTime]|[cpuUsage]|[peakCpuUsage]|[userUsage]|[peakUserUsage]|[label]
port|[portType]|[index]|[avgBytes]|[avgRows]|[peakBytes]|[peakRows]|[totalBytes]|[totalRows]

    (1)

    CloverDX no longer stores graphVersion, so the value is null.

    example of request

    http://localhost:8080/clover/simpleHttpApi/executions_history?from=&to=2008-09-16+16%3A40&status=&sandbox=def&graphID=&index=&records=&returnType=DESCRIPTION

    example of DESCRIPTION (plain text) response 

    execution|13108|FINISHED_OK|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:58|nodeA|2.4
phase|0|38733
node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0|label_text
port|Output|0|0|0|0|0|130|10
node|TRASH0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0|label_text
port|Input|0|0|0|5|0|130|10
node|SPEED_LIMITER0|FINISHED_OK|0|0|0.0|0.0|0.0|0.0|label_text
port|Input|0|0|0|0|0|130|10
port|Output|0|0|0|5|0|130|10
execution|13107|ABORTED|clover|def|test.grf|2008-09-16 11:11:19|2008-09-16 11:11:30
phase|0|11133
node|DATA_GENERATOR1|FINISHED_OK|0|0|0.0|0.0|0.0|0.0|label_text
port|Output|0|0|0|0|0|130|10
node|TRASH0|RUNNING|0|0|0.0|0.0|0.0|0.0|label_text
port|Input|0|5|0|5|0|52|4
node|SPEED_LIMITER0|RUNNING|0|0|0.0|0.0|0.0|0.0|label_text
port|Input|0|0|0|0|0|130|10
port|Output|0|5|0|5|0|52|4

    For returnType==DESCRIPTION_XML returns a complex data structure describing one or more selected executions in XML format. A schema describing XML format of the XML response is accessible on CloverDX Server URL: http://[host]:[port]/clover/schemas/executions.xsd

    Operation suspend

    Suspends the Server or sandbox (if specified). No graphs may be executed on suspended Server/sandbox.

    parameters

    Table 37.6. Parameters of suspend

    NameMandatoryDefaultDescription
    sandboxno-

    The text ID of a sandbox to suspend. If not specified, it suspends the whole Server.

    atonceno 

    If this parameter is set to true, running graphs from suspended Server (or just from sandbox) are aborted. Otherwise it can run until it is finished in standard way.


    returns

    Result message

    Operation resume

    parameters

    Table 37.7. Parameters of resume

    NameMandatoryDefaultDescription
    sandboxno-

    The text Id of a sandbox to resume. If not specified, the Server will be resumed.

    verbosenoMESSAGEMESSAGE | FULL - how verbose should the possible error message be.

    returns

    Result message

    Operation sandbox_create

    This operation creates a specified sandbox. If it is a sandbox of "partitioned" or "local" type, it also creates locations by "sandbox_add_location" operation.

    parameters

    Table 37.8. Parameters of sandbox create

    NameMandatoryDefaultDescription
    sandboxyes-The text ID of a sandbox to be created.
    pathno-A path to the sandbox root if the Server is running in a standalone mode.
    typenoshared

    Sandbox type: shared | partitioned | local. For a standalone Server may be left empty, since the default "shared" is used.

    createDirsnotrue

    Switch whether to create a directory structure of the sandbox (only for a standalone Server or "shared" sandboxes in a Cluster environment).

    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    Result message

    Operation sandbox_add_location

    This operation adds a location to the specified sandbox. Can be only used with partitioned or local sandboxes.

    parameters

    Table 37.9. Parameters of sandbox add location

    NameMandatoryDefaultDescription
    sandboxyes-A sandbox which we want to add a location to.
    nodeIdyes-A location attribute - a node which has direct access to the location.
    pathyes-A location attribute - a path to the location root on the specified node.
    locationno-A location attribute - a location storage ID. If not specified, a new one will be generated.
    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    Result message

    Operation sandbox_remove_location

    This operation removes a location from the specified sandbox. Only sandboxes of the partitioned or local type can have locations associated.

    parameters

    Table 37.10. Parameters of sandbox add location

    NameMandatoryDefaultDescription
    sandboxyes-Removes a specified location from its sandbox.
    locationyes-

    A location storage ID. If the specified location isn't attached to the specified sandbox, the sandbox won't be changed.

    verbosenoMESSAGEMESSAGE | FULL - how verbose should possible error message be.

    returns

    Result message

    Operation download_sandbox_zip

    This operation downloads the content of a specified sandbox as a ZIP archive.

    parameters

    Table 37.11. Parameters

    NameMandatoryDefaultDescription
    sandboxyes-A code of the sandbox to be downloaded.

    returns

    a content of a specified sandbox as a ZIP archive

    example

     wget --http-user=username --http-password=password http://localhost:8080/clover/simpleHttpApi/download_sandbox_zip?sandbox=my-sandbox

    Operation upload_sandbox_zip

    This operation uploads the content of a ZIP archive into a specified sandbox.

    parameters

    Table 37.12. Parameters

    NameMandatoryDefaultDescription
    sandboxyes-A code of the sandbox the ZIP file will be expanded to.
    zipFileyes-The ZIP archive file.
    overwriteExistingnofalseIf true, the files already present in the sandbox will be overwritten.
    deleteMissingnofalseIf true, the files not present in the ZIP file will be deleted from the sandbox.
    fileNameEncodingnoUTF-8The encoding that was used to store file names in the ZIP archive.

    returns

    Result message

    an example of request (with using curl CLI tool (http://curl.haxx.se/)) 

    curl -u username:password -F "overwriteExisting=true"
    -F "zipFile=@/tmp/my-sandbox.zip"
    http://localhost:8080/clover/simpleHttpApi/upload_sandbox_zip

    Operation cluster_status

    This operation displays Cluster's nodes list.

    parameters

    no

    returns

    A list of Cluster nodes with information in the following format:

    <Node Name>|<Node HTTP URL>|<System Load Average>|<Node Status>

    example

    node01|http://localhost:8083/clover|0.3|READY

    Note: The value of system load average is calculated from the minute preceding the call. If it cannot be obtained, a negative value is returned (may be caused by the calculation's unacceptable performance impact or lack of support by operating system).

    Operation export_server_config

    This operation exports a current server configuration in XML format.

    parameters

    Table 37.13. Parameters of server configuration export

    NameMandatoryDefaultDescription
    includenoall

    Selection of items that will be included in the exported XML file; the parameter may be specified multiple times. Possible values are:

    • all - include items of all types
    • users - include a list of users
    • userGroups - include a list of user groups
    • sandboxes - include a list of sandboxes
    • jobConfigs - include a list of job configuration parameters
    • schedules - include a list of schedules
    • eventListeners - include a list of event listeners
    • launchServices - include a list of launch services (deprecated)
    • tempSpaces - include a list of temporary spaces

    returns

    Current server configuration as an XML file.

    example

    wget --http-user=username --http-password=password http://localhost:8080/clover/simpleHttpApi/export_server_config

    Operation import_server_config

    This operation imports server configuration.

    parameters

    Table 37.14. Parameters of server configuration import

    NameMandatoryDefaultDescription
    xmlFileyes-An XML file with server's configuration.
    dryRunnotrueIf true, a dry run is performed with no actual changes written.
    verbosenoMESSAGE

    MESSAGE | FULL - how verbose should the response be: MESSAGE for a simple message, FULL for a full XML report.

    newOnlynofalse

    If true only new items will be imported to the Server; the items already present on the Server will be left untouched.

    overridenofalse

    If override is set to true, the import override existing configuration and the newOnly parameter is ignored. The parameter has an effect only if the configuration was exported from CloverDX Server 5.5.0 or newer. It doesn't affect user and user groups configuration.

    includenoall

    Selection of items that will be imported from the XML; the parameter may be specified multiple times. Possible values are:

    • all - import items of all types
    • users - import users
    • userGroups - import user groups
    • sandboxes - import sandboxes
    • jobConfigs - import job configuration parameters
    • schedules - import schedules
    • eventListeners - import listeners
    • launchServices - import launch services (deprecated)
    • tempSpaces - import temporary spaces

    returns

    Result message or XML report

    an example of request (with using curl CLI tool (http://curl.haxx.se/)) 

    curl -u username:password -F "dryRun=true" -F "verbose=FULL"
    -F "xmlFile=@/tmp/clover_configuration_2013-07-10_14-03-23+0200.xml"
    http://localhost:8080/clover/simpleHttpApi/import_server_config