Ratchet

Appearance

Execution Model ​

This page explains how Ratchet moves a job from "sitting in the database" to "running on a worker thread" and back. The key components are the Poller, the JobExecutionCoordinator, and the JobTask.

Execution Pipeline ​

The flow for each poll cycle:

  1. Poll: The Poller queries for eligible jobs using SELECT ... FOR UPDATE SKIP LOCKED
  2. Claim: Matching jobs are atomically transitioned from PENDING to RUNNING
  3. Dispatch: Each claimed job is submitted to the JobExecutionCoordinator thread pool
  4. Execute: A JobTask validates, runs, and handles the result on a worker thread
  5. Post-process: Based on outcome and job type, the engine triggers chains, workflows, or batch updates

The Poller ​

The Poller is the heart of Ratchet's pull-based architecture. It periodically queries the database for jobs that are ready to execute.

Claim Query ​

The core query selects PENDING jobs whose scheduled time has passed, ordered by effective priority and due time:

sql
SELECT job_id FROM scheduler_job_queue
WHERE status = 'PENDING'
  AND scheduled_time <= CURRENT_TIMESTAMP
ORDER BY (priority + age_boost) DESC, scheduled_time ASC
FOR UPDATE SKIP LOCKED
LIMIT :batchSize

Key properties of this query:

  • FOR UPDATE SKIP LOCKED -- Two nodes running this query concurrently will claim disjoint sets of jobs. Neither blocks the other. This is the foundation of Ratchet's cluster-safe execution.
  • Priority ordering -- Raw priority is the base priority.
  • Age boosting -- age_boost is computed as floor(wait_minutes / priorityBoostIntervalMinutes), so sufficiently old low-priority work can move ahead of newer high-priority work. Configure the interval with RatchetOptions.store(...).
  • Age ordering -- Within the same effective priority, older jobs are claimed first (fairness).
  • Batch claiming -- Up to batchSize jobs are claimed in a single query to reduce database round trips.

After the query, claimed jobs are updated:

  • status = RUNNING
  • picked_by = current node ID
  • picked_at = current timestamp

Adaptive Polling ​

The Poller doesn't use a fixed interval. The PollingStrategy dynamically adjusts the delay between polls:

ModeDelayTrigger
Burst500msWakeup signal received
Normal2-30sSteady-state operation
Deep Idle60sNo jobs found for 5+ minutes

The algorithm considers:

  • Rolling window of the last 10 poll results to identify trends
  • Full batch detection -- when 3+ consecutive polls return a full batch (all batchSize slots filled), polling becomes very aggressive (delay / 4)
  • System load factor -- thread pool utilization adjusts the delay. High utilization slows polling; low utilization allows faster polling
  • Idle count -- consecutive empty polls trigger exponential backoff
  • Deep idle threshold -- after 5 minutes without finding any jobs, the poller enters deep idle (60s delay)

When a wakeup signal arrives (from ClusterCoordinator.notifyNewWork() or a local job submission), the poller immediately exits deep idle and enters burst mode.

Drain Mode ​

During CDI shutdown, an internal DrainController makes subsequent poll cycles skip new claims before the Poller stops. The execution coordinator then requests cancellation of active work; unfinished RUNNING rows remain durable and are handled by normal shutdown reset or orphan recovery.

DrainController is not a public health or lifecycle API. An application's readiness endpoint must own its traffic-acceptance state rather than inject an RI implementation type. See the Kubernetes rolling-termination pattern.

Job Execution Coordinator ​

The JobExecutionCoordinator manages the thread pools that execute jobs. When the Poller claims a batch of jobs, each job is submitted to the coordinator for asynchronous execution.

Thread Pool Management ​

The ThreadPoolManager maintains one or more thread pools for job execution. The ExecutorProvider SPI controls how these pools are created:

  • Default (CDI): Uses @Resource-based container-managed executors (e.g., WildFly's ManagedExecutorService)
  • Custom: Implement ExecutorProvider to use virtual threads, custom pool sizes, or application-specific executors

The coordinator tracks pool utilization metrics, which feed back into the PollingStrategy to adjust polling aggressiveness.

Lazy Entity Loading ​

For efficiency, the Poller passes lightweight JobClaimDto objects (containing only the job ID and essential metadata) to the coordinator rather than full JobEntity objects. The JobTask loads the full entity from the database only when execution begins. This avoids holding large entity graphs in memory between claim and execution.

JobTask: The Worker ​

JobTask implements Callable<Void> and encapsulates the complete execution lifecycle for a single job. Each instance is dedicated to one job and discarded after completion.

Execution Steps ​

Setup MDC contextAdds job ID and node ID so logs can be correlated.
Load `JobEntity`Loads the full entity lazily from the lightweight `JobClaimDto`.
Bind `JobContext`Attaches logger, params, and any delivered signal payload to the worker thread.
Check cancellationRe-reads status and exits cleanly if the job was canceled after claim.
Check circuit breakerAsks `ResilienceStrategy` whether the target service is available.
Acquire resource permitReschedules without consuming a retry if the configured resource is saturated.
Validate class policyApplies `ClassPolicy` before resolving the target invocation.
Resolve bean and methodUses CDI lookup and reflection from the persisted payload.
Execute targetRuns through `ResilienceStrategy`, including circuit-breaker wrapping.
Handle resultRoutes success to `handleSuccess()` or failure to `handleFailure()`.
Release resource permitFrees capacity for other queued work.
Clear MDC and contextRemoves thread-local state before the worker returns to the pool.

Step Details ​

1-3. Context Setup

The JobMdcContext sets up SLF4J/JUL MDC entries (job ID, node ID) so all log output during execution is automatically correlated. JobContext.bind() attaches the thread-local context that jobs can access via JobContext.current().

4. Cancellation Check

Before doing any work, the task re-reads the job's status from the database. If the job was canceled between claim and execution, results are discarded and the execution ends cleanly.

5. Circuit Breaker Gate

The ResilienceStrategy SPI is consulted to check if the target service is available. If the circuit breaker is OPEN, the job is rescheduled with a delay (matching the circuit breaker's OPEN-to-HALF_OPEN transition window) without counting as a retry attempt.

6. Resource Permit

If the job specifies a resourceName, the ResourcePermitService attempts to acquire a permit. If the resource is at capacity, the job is rescheduled with a configurable delay. This is not a failure; no retry is consumed.

7. Security Validation

The PreExecutionValidator invokes the ClassPolicy SPI to verify the target class is allowed. The default PackagePrefixClassPolicy is deny-all until you provide an @Alternative bean with your application's package prefixes, and RatchetProducer fails deployment by default so this misconfiguration is caught at startup.

8. Bean Resolution

For instance methods, the BeanResolver SPI locates the CDI bean for the target class. For static methods, no bean is needed. Method resolution uses ASM descriptors to match the exact method signature, with results cached in a ConcurrentHashMap.

9. Execution

The actual method invocation happens through ResilienceStrategy.execute(), which wraps the call in circuit breaker protection. The method is invoked reflectively with the deserialized arguments.

10. Result Handling

On success:

  • Execution timing is recorded
  • Return value is serialized to JSON
  • Status transitions RUNNING -> SUCCEEDED
  • JobCompletedEvent is published
  • PostExecutionHandler.handleJobSuccess() routes to batch/chain/workflow handlers

On failure:

  • Attempt counter is incremented atomically
  • @DoNotRetry is checked on the exception
  • RetryPolicy.shouldRetry() is consulted
  • Either schedules a retry or moves to DLQ

Mid-Execution Cancellation ​

The executor checks wasJobCanceledDuringExecution() both before starting work and after the method returns. If the job was canceled during execution, the result is discarded and the job stays in CANCELED state. This handles the race condition where cancelJob() is called while a job is actively running.

Transactions and Database Connections ​

JobTask.call() does not run inside a transaction. This is deliberate: holding a database connection open for the entire duration of a long-running job would exhaust connection pools.

Instead, each database operation (status update, retry scheduling, success marking) is a separate short transaction. The @Transactional annotation is on the individual service methods (PostExecutionHandler, DeadLetterService, etc.), not on JobTask itself.

Timeout Handling ​

Each execution attempt has a deadline: the time the attempt started plus the job's timeout. A job without its own timeout uses ratchet.timeout.default-sla-seconds (30 minutes by default). A watchdog on the executing node does up to three things for each attempt. When the grace is longer than the soft-warning margin, step 2 happens before step 1.

  1. At ratchet.timeout.soft-timeout-percent of the timeout (80% by default), it logs a warning.
  2. At the deadline minus ratchet.timeout.cancellation-grace-seconds, it sets the attempt's cancellation request. This step is skipped when the grace is 0 (the default) or not shorter than the timeout.
  3. At the deadline, if the attempt is still running, it sets the cancellation request (if it is not set yet) and interrupts the worker thread with Future.cancel(true). The attempt then counts as a timeout: it is retried if it has retries left, otherwise it fails and goes to the DLQ. An attempt that already stopped after the request is not interrupted (see Stopping at a safe point).
java
scheduler.enqueue(() -> longRunningTask())
    .withTimeout(Duration.ofMinutes(5))
    .submit();

Stopping at a safe point ​

An interrupt lands wherever the worker happens to be blocked: waiting for a pooled bean, a JDBC connection, or a cache lock. The code it interrupts may report the timeout as an unrelated error and leave container state half-changed. A job that works in units, such as pages of a table, can stop between units instead. JobContext tells it when to stop:

MethodReturns
deadline()Optional<Instant>: when the watchdog will interrupt this attempt. Empty when the context was not bound for a watched execution, for example inside an onFailure callback that the watchdog runs after it interrupts a job.
isCancellationRequested()true once Ratchet has asked this attempt to stop. Cheap to call; it does no I/O.
throwIfCancellationRequested()Throws CancellationRequestedException if a stop was requested.
java
public void migrate() {
  JobContext ctx = JobContext.current();
  while (hasMorePages()) {
    ctx.throwIfCancellationRequested(); // stop between pages, not in the middle of one
    copyNextPage();
  }
}

Set a grace period that is longer than one unit of work, so the job sees the request and finishes its current unit before the deadline:

properties
ratchet.timeout.cancellation-grace-seconds=60

What happens next depends on how the job ends:

  • It throws CancellationRequestedException (directly, or as the cause of another exception) and the exception was created after the request: the attempt is a timeout, exactly as if the watchdog had interrupted it. The attempt count, retry backoff, JobExecutionTimedOutEvent, DLQ routing and onFailure callback are the same. The worker thread is not interrupted. The exception records the request state when it is created, so one created before the request stays an ordinary failure, even if the request arrives while it unwinds through a finally block or a wrapper. The exception also records the attempt it was created in, so one kept from an earlier attempt and rethrown by a retry is an ordinary failure. throwIfCancellationRequested() only throws after a request, so its exception counts for that attempt. If you create the exception yourself on another thread, build it with CancellationRequestedException.forContext(message, context) and pass the job's JobContext, because the context is only bound on the job thread.
  • It returns normally: the attempt succeeds. Ratchet cannot tell a job that stopped early from one that finished just before its deadline, so a job that stops before its work is done must throw.
  • It keeps running: the watchdog interrupts it at the deadline, as before.

Each attempt has its own request flag, so a retry starts with the flag cleared. The flag is held in memory on the node running the attempt, where the watchdog also runs.

Cancelling a running job with JobSchedulerService.cancelJob does not set the request yet. The job's status changes to CANCELED in the store, and the worker discards the result when the job returns.

Performance Characteristics ​

Method Caching ​

JobTask maintains static ConcurrentHashMap caches for:

  • Class lookups (Class.forName results)
  • Method resolution (reflected Method objects)
  • Service name resolution (circuit breaker service names)

These caches are cleared on application shutdown via JobTask.clearCaches() to prevent classloader leaks in redeployable containers.

Heartbeat ​

The DynamicHeartbeatCalculator adjusts the heartbeat interval based on system load. Heartbeats update the node's heartbeat_ts value in the scheduler_node table, so orphan recovery can detect crashed nodes.