Chapter 32. Job Queue

Job execution request - request to start a job. The request can come from users in Server Console, Designer, API, CloverDX internal components etc.

Job execution command - information about which job is to be executed, its configuration (e.g. graph parameters) and configuration of its environment (e.g. log level).

Job queue - the core of the Job Queue. When it receives a job execution command, it decides whether the job can be started immediately or whether it will be enqueued. The job queue regularly checks load metrics and jobs in the queue to determine which jobs can be taken from the queue and started.

Load metrics - metrics about system CPU load and heap memory usage of CloverDX Server and Worker.

Job execution - starts and runs a job

Submitted - time when the request to start the job was received (regardless of whether the job was enqueued or not).

Started - time when the job was actually started. If the job was enqueued, this is the time the job was taken from the queue and started its execution. Until the job is actually started, the Started time is empty. If the job was not enqueued but it was immediately started, then the Started time is the same as Submitted time.

System CPU load - total CPU load of the system, includes all processes running on the system - CloverDX Server Core, Worker and all other processes. A CPU intensive process running outside of CloverDX Server can cause high system CPU load.

Core heap memory usage - used Java heap memory in CloverDX Server Core. This memory is mostly used by internal Server logic.

Worker heap memory usage - used Java heap memory in CloverDX Server Worker. This memory is mostly used by running jobs.

When a job execution command is received by the queue, the queue determines whether the job will be started immediately, or enqueued. The decision is based on load metrics:

The job queue regularly (10 times per second, see jobqueue.checkInterval configuration property) checks the current load and based on that it can start some of the enqueued jobs:

Data Services (and Data Apps) - Data Services are not processed by the job queue by default - enqueued Data Services might cause timeouts on the clients calling them. It is possible to enable enqueueing of all Data Services (via the jobqueue.dataservice.enabled configuration property), e.g. when using Data Services for orchestration of job executions. It is also possible to enable enqueuing of specific Data Services via the queueable execution property.

Profiler jobs - job queue does not support Profiler jobs (*.cpj files), they are started immediately.

Event Listeners and Schedules:.

max_running_concurrently and enqueue_executions execution properties - functionality of these properties is kept, and they are evaluated after the job queue for now. So the job queue enqueues jobs and starts their execution without being affected by these execution properties. When the job startup begins after the queue, Server makes sure these execution properties are respected.

APIs - clients of CloverDX Server’s APIs must be aware that jobs can be enqueued:

Cluster - each node of the cluster has its own job queue. Cluster load balancer is aware of the job queues of all cluster nodes. When the load balancer decides on which cluster node a job will be executed, it takes job queue size into account. The load balancer prioritizes nodes that have an empty job queue (i.e. nodes that have low load and don’t need to enqueue jobs). Nodes that have an empty job queue ingore the cluster.lb.memory.limit and cluster.lb.cpu.limit configuration properties - protection from high CPU and heap memory usage is provided by the queue, see Jobs Load Balancing Properties.

Cluster job partitions - job in cluster can be partitioned to run on multiple nodes (see Chapter 49, Sandboxes in Cluster). Each of the partitions can be enqueued, while the master job that orchestrates them is never partitioned.

Server suspend - when suspending the server (see Suspending the Server), all currently running and enqueued jobs are left to finish their execution. When using the suspend at once action, all running and enqueued jobs are immediately aborted.

Job does not start immediately - when running a job (e.g. from Server Console), the job can be enqueued and start later. Job queue enqueues a job in case of high load or emergency mode - look in performance graphs in Monitoring, queue log and performance log. Enqueued jobs are started in order based on Submit time, so executions requested earlier are typically started earlier too.

Enqueued job is needed to start immediately - if a job is enqueued and you need it to start immediately, e.g. to meet a data processing deadline, then you can use the Start Enqueued Job Immediately action to manually force the job to start.

Job launched by schedule or event listener doesn’t start immediately - jobs launched by schedules and even listeners can be also enqueued, e.g. a job can start later than its scheduled start time. You can disable enqueuing on a job by the queueable execution property.

Subjobs must run in parallel - if some subjobs of a parent job must run in parallel, the job queue could break this condition. In such a case you can disable enqueuing on the subjobs by the queueable execution property.

Jobs are enqueued even if CPU load is low - job queue can be in Emergency mode in case of high heap memory usage, see Job Queue Emergency Mode.

High number of enqueued jobs - the job queue is probably enqueuing jobs because of high CPU or heap memory load - look in performance graphs in Monitoring, queue log and performance log.

Increasing number of enqueued jobs - if the number of enqueued jobs continues to increase, this means that the server keeps getting more job execution requests than it can process. The job queue protects the server from overload. In this case investigate who is sending the job execution requests, e.g. client side of the server APIs.

Maximum size of job queue - the number of jobs that can be enqueued is limited (by default the limit is 100 000, configurable via jobqueue.maxQueueSize configuration property). If the limit is reached, job executions fail with “Maximum size of job queue was reached. You may want to consider increasing CloverDX Server property "jobqueue.maxSize". Job execution with run ID 140000 aborted at 23:05:23” error message.

jobqueue.enabled=true - set to false to disable job queue globally. If the job queue is disabled, all jobs are started immediately when their execution request arrives.

jobqueue.dataservice.enabled=false - by default Data Services (and Data Apps) are not processed by the job queue, but started immediately (see Data Services and job queue). You can enable enqueueing of Data Services via this property.

jobqueue.systemCpuLimit=0.85 - system CPU load (0.85 is 85%) which is considered high and causes jobs to be enqueued.

jobqueue.coreHeapUsageLimit=0.90 - usage of CloverDX Server Core heap memory (0.90 is 90%), above which the job queue enters emergency mode.

jobqueue.workerHeapUsageLimit=0.90 - usage of CloverDX Server Worker heap memory (0.90 is 90%), above which the job queue enters emergency mode