Cluster Configuration

Ratchet supports multi-node deployments where multiple application instances share the same database. This guide covers how to configure clustering for safe job claiming, recurring job deduplication, and node coordination.

How clustering works

In a clustered Ratchet deployment:

1. Job claiming is safe by default. The database handles concurrency via row-level locking (SELECT ... FOR UPDATE SKIP LOCKED on PostgreSQL and Oracle, InnoDB row locks on MySQL, UPDLOCK, READPAST row locks on SQL Server, atomic updates on MongoDB). No additional configuration is needed for one-shot jobs.

2. Recurring execution and startup coordination are separate concerns. Runtime recurring execution uses scheduler_lock so only one node advances the recurring scheduler at a time, and destructive startup cleanup is gated by the built-in StartupCoordinator, which also uses store-backed leases.

3. Node identity is tracked. Each node registers itself in the scheduler_node table with a heartbeat. This enables distributed locking and stale-node detection.

Enabling cluster mode

There is no separate ratchet.cluster.enabled switch. A deployment becomes clustered when multiple Ratchet nodes share the same store and you provide the coordination pieces you need. In practice that means:

• Node heartbeat registration in scheduler_node
• Distributed lock acquisition via scheduler_lock
• Singleton recurring-scheduler execution via scheduler_lock
• Store-backed startup leases for destructive initialization work
• Cross-node wakeup notifications (if a ClusterCoordinator is provided)

Node identity

Each node in the cluster needs a unique, stable identifier. Implement the NodeIdentityProvider SPI:

@ApplicationScoped
public class KubernetesNodeProvider implements NodeIdentityProvider {

  @Override
  public String getNodeId() {
    // Kubernetes pod name is stable within a StatefulSet
    String podName = System.getenv("HOSTNAME");
    return podName != null ? podName : InetAddress.getLocalHost().getHostName();
  }
}

The default implementation uses hostname-PID-<8-char-UUID>. Override it when:

• Running in Kubernetes (use the pod name from metadata.name)
• Running in Docker Compose (use the HOSTNAME environment variable)
• Running multiple instances on the same host (use a unique port or instance ID)

There is no built-in ratchet.cluster.node-id property. Override node identity by providing your own NodeIdentityProvider bean when the hostname-based default is not sufficient.

ClusterCoordinator SPI

The ClusterCoordinator interface coordinates cross-node wakeup notifications:

@Incubating
public interface ClusterCoordinator extends AutoCloseable {

  /**
   * Notifies the cluster that new work is available at the given priority level.
   * Other nodes can use this signal to wake up their polling engines immediately.
   * {@code source} identifies the originating node; {@code executionTarget} is an
   * informational routing label, or null when the wakeup is not target-scoped.
   */
  void notifyNewWork(JobPriority priority, NodeIdentity source, String executionTarget);

  /**
   * Registers a listener that is called when another node signals new work.
   * The polling engine uses this to wake up and check for available jobs.
   */
  void registerWakeupListener(Consumer<JobWakeupHint> listener);

  /** Releases transport resources held by this coordinator. Must be idempotent. */
  void close();
}

Database-only coordination

If you do not provide a ClusterCoordinator, Ratchet still coordinates one-shot claims, recurring scheduler execution, and destructive startup cleanup through the store. Each node simply polls independently, so wakeups are slower.

First-party coordinator modules

You usually do not implement this SPI yourself. Ratchet ships four coordinator modules (PostgreSQL LISTEN/NOTIFY, JMS, Hazelcast, and Infinispan) that you enable by adding a dependency, with no beans.xml change. See Cluster Coordinators for setup, configuration, delivery guarantees, failure behavior, and metrics. The examples below show how to build a custom transport when none of the shipped modules fit.

StartupCoordinator

StartupCoordinator gates destructive startup work behind a store-backed lease:

@Incubating
public interface StartupCoordinator {
  boolean tryAcquire(String actionName, Duration leaseTtl);
  void release(String actionName);
}

The default StoreBackedStartupCoordinator uses scheduler_lock, so no extra cluster configuration is required for recurring-annotation orphan cleanup.

Redis-based coordinator

A Redis-based implementation for low-latency cross-node notifications:

@ApplicationScoped
public class RedisClusterCoordinator implements ClusterCoordinator {

  @Inject
  RedisClient redis;

  private final List<Consumer<JobWakeupHint>> listeners = new CopyOnWriteArrayList<>();

  @PostConstruct
  void init() {
    // Subscribe to wakeup channel in a background thread
    redis.subscribe("ratchet:wakeup", hint -> {
      listeners.forEach(l -> l.accept(hint));
    });
  }

  @Override
  public void notifyNewWork(JobPriority priority, NodeIdentity source, String executionTarget) {
    redis.publish("ratchet:wakeup", priority.name());
  }

  @Override
  public void registerWakeupListener(Consumer<JobWakeupHint> listener) {
    listeners.add(listener);
  }

  @Override
  public void close() {
    redis.close();
  }
}

JMS/messaging-based coordinator

For environments that already have a message broker, use the first-party ratchet-coordinator-jms module rather than building this by hand. If you need a transport it does not cover, implement ClusterCoordinator over a shared topic and invoke the registered listeners from the container-managed consumer when a wakeup signal arrives.

Distributed locking

Ratchet uses the scheduler_lock table for distributed coordination:

-- PostgreSQL schema (created by the DDL)
CREATE TABLE IF NOT EXISTS scheduler_lock (
    lock_name  VARCHAR(128) NOT NULL,
    owner_node VARCHAR(64)  NOT NULL,
    locked_at  TIMESTAMPTZ(6) NOT NULL,
    expires_at TIMESTAMPTZ(6) NOT NULL,
    CONSTRAINT pk_scheduler_lock PRIMARY KEY (lock_name)
);

Locks are acquired with an expiration time. If a node crashes, its locks expire and another node can acquire them. This prevents deadlocks from node failures.

Singleton lease lifetimes

Each scan closes its lease when it finishes. The TTL limits how long a crashed holder blocks another node.

If a node dies during its own orphan scan, another node can release the orphan recovery lease before its TTL runs out. It does so once the dead node's heartbeat is older than ratchet.node.orphan-grace-seconds, then runs the scan. With the defaults (60-second grace, 30-second TTL, heartbeats at most 20 seconds apart) the TTL usually runs out first, so the early release matters when you raise the TTL or a node's heartbeats are delayed. The other leases always wait for their TTL.

Singleton lease	TTL property	Default
orphanRecovery	ratchet.node.orphan-recovery-lease-ttl-seconds	30 seconds
recurringScheduler	ratchet.recurring.lease-ttl-seconds	300 seconds
batchRecovery	ratchet.batch.recovery-lease-ttl-seconds	900 seconds
signalTimeoutScan	ratchet.timeout.signal-timeout-lease-ttl-seconds	120 seconds
dlqPurger	Fixed TTL	600 seconds
jobArchiver	Fixed TTL	7200 seconds
logPurger	Fixed TTL	600 seconds

Orphan scans run every ratchet.node.orphan-scan-interval-seconds (default 300). Batch recovery runs every ratchet.batch.recovery-interval-seconds (default 900). Its first scan waits at most 60 seconds, or one interval if shorter. For a test cluster, lower the scan intervals, lease TTLs, and orphan grace together. Recurring leases require at least 2 seconds. The other configurable TTLs and scan intervals require at least 1 second.

How recurring job deduplication works

Ratchet uses a cluster-wide singleton lease named recurringScheduler to ensure exactly one node runs the recurring scheduler at a time:

1. On each poll cycle, each node tries to acquire the recurringScheduler singleton lease from scheduler_lock
2. Only the node that holds the lease scans scheduler_recurring_job for due next_fire rows and spawns child jobs
3. Nodes that fail to acquire the lease skip the cycle and retry on the next poll interval
4. The lease renews every max(1, leaseTtlSeconds * 2 / 5) seconds while held (120 seconds by default); if the lease-holding node crashes, the lease expires and another node acquires it on its next cycle

Node heartbeats

Each node maintains a heartbeat in the scheduler_node table:

-- Populated automatically by the scheduler's heartbeat on each node
SELECT * FROM scheduler_node;

-- Example output:
-- node_id          | heartbeat_ts             | started_at
-- ratchet-node-0   | 2026-03-31 10:00:05.123  | 2026-03-31 08:00:00.000
-- ratchet-node-1   | 2026-03-31 10:00:04.456  | 2026-03-31 08:00:01.000
-- ratchet-node-2   | 2026-03-31 10:00:05.789  | 2026-03-31 08:00:02.000

Heartbeats are used to:

• Detect stale nodes that may have crashed
• Determine cluster size for scaling decisions
• Identify which node is executing which jobs (via picked_by)

Stale node detection

A node is stale when its heartbeat exceeds the configured threshold. Jobs that node was running will time out and become eligible for retry by other nodes.

-- Find stale nodes (no heartbeat in 60 seconds)
SELECT node_id, heartbeat_ts
FROM scheduler_node
WHERE heartbeat_ts < NOW() - INTERVAL '60 seconds';

Kubernetes-specific configuration

StatefulSet with cluster mode

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ratchet-scheduler
spec:
  serviceName: ratchet-scheduler
  replicas: 3
  selector:
    matchLabels:
      app: ratchet-scheduler
  template:
    metadata:
      labels:
        app: ratchet-scheduler
    spec:
      containers:
      - name: app
        image: myapp:latest
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

Pod names (ratchet-scheduler-0, ratchet-scheduler-1, ratchet-scheduler-2) are used as stable node identifiers.

Scaling considerations

• Scaling up: New nodes start polling and claiming jobs immediately. No manual intervention needed.
• Scaling down: Use the Kubernetes rolling-termination pattern so request traffic leaves the pod before its Jakarta runtime shuts down. Size terminationGracePeriodSeconds for the preStop delay plus container/CDI teardown; Ratchet may cancel active jobs and recover them on another node rather than waiting for arbitrary job runtimes.
• Pod disruption: Use a PodDisruptionBudget to prevent too many nodes from being evicted simultaneously.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ratchet-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: ratchet-scheduler

Failure modes

Node failure

If a node crashes while executing jobs:

1. The node's heartbeat stops updating in scheduler_node
2. Jobs it was executing remain in RUNNING status
3. When the job timeout expires, jobs are marked FAILED
4. If retries are configured, failed jobs become PENDING and are claimed by surviving nodes
5. Distributed locks held by the dead node expire, allowing other nodes to acquire them

Network partition

In a network split:

• Each partition continues executing one-shot jobs independently (safe, because the database prevents double-claiming)
• Recurring job scheduling may temporarily pause for partitions separated from the database, since the singleton lease cannot be renewed
• After the partition heals, the system self-corrects: lease expiration and idempotency keys prevent lasting inconsistency

Split-brain prevention

Ratchet uses the database as the single source of truth. All coordination goes through the database (or through ClusterCoordinator for notifications only). Split-brain is limited to the notification layer; the database always arbitrates who executes what.

Monitoring cluster health

Active nodes

SELECT node_id, heartbeat_ts, started_at
FROM scheduler_node
WHERE heartbeat_ts > NOW() - INTERVAL '30 seconds'
ORDER BY started_at;

Jobs per node

-- picked_by / RUNNING are live state on scheduler_job_queue.
SELECT picked_by AS node, COUNT(*) AS running_jobs
FROM scheduler_job_queue
WHERE status = 'RUNNING'
GROUP BY picked_by;

Lock status

SELECT lock_name, owner_node, locked_at, expires_at
FROM scheduler_lock
WHERE expires_at > NOW()
ORDER BY locked_at;

See also

• Deployment Overview -- General deployment guidance
• Kubernetes Deployment -- StatefulSet and pod configuration
• Performance Tuning -- Tuning polling and thread pools
• Clustering & Distributed Execution -- Conceptual overview of clustering
• Monitoring & Observability -- Metrics and health checks