Graceful Shutdown for Workers

When Ryvn performs a rolling deploy or scales down an installation, it sends a SIGTERM signal to each instance being replaced. If your application doesn’t handle this signal, the process is force-killed after the termination grace period — any in-flight work (running workflows, processing queue messages, active requests) is lost. This guide walks through configuring graceful shutdown so your workers finish their current work before exiting. While the examples use Temporal workers, the same pattern applies to any long-running process — RabbitMQ consumers, background job processors, or custom queue workers.

How it works

During a rolling deploy, Ryvn starts the new instance first. Once it’s ready, Ryvn sends SIGTERM to the old instance. Your worker catches that signal, stops polling for new tasks, and drains any in-flight work before exiting cleanly. Ryvn waits up to terminationGracePeriodSeconds before force-killing the instance.

Step 1: Handle SIGTERM in your worker

Your application needs to listen for SIGTERM and initiate a clean shutdown. The key behavior is:

Stop accepting new work — stop polling queues or accepting new requests
Finish in-flight work — let running tasks, workflows, or activities complete
Exit cleanly — exit the process once all work is drained

Temporal (TypeScript)
Generic Node.js
Go

Temporal’s Worker class has a built-in shutdown() method that stops the worker from picking up new tasks and allows in-flight workflows and activities to complete. The worker.run() promise resolves once the worker has fully drained.

worker.ts

import { Worker } from '@temporalio/worker';

async function main() {
  const worker = await Worker.create({
    // ... your worker config
    taskQueue: 'my-queue',
    shutdownGraceTime: '30s',
  });

  // Trigger shutdown on SIGTERM / SIGINT
  const shutdown = () => worker.shutdown();
  process.on('SIGTERM', shutdown);
  process.on('SIGINT', shutdown);

  // worker.run() resolves once shutdown completes and all work is drained
  await worker.run();

  // All in-flight work is done — exit cleanly
  process.exit(0);
}

main().catch((err) => {
  console.error('Worker failed', err);
  process.exit(1);
});

worker.shutdown() is a void method that signals the worker to stop. It does not return a promise. The worker.run() promise resolves once shutdown is complete and all in-flight work has drained.

For any Node.js process that does long-running work, you can use the same pattern:

worker.ts

let isShuttingDown = false;
let activeJobs = 0;

// Your job processing loop — track in-flight work
async function processNextJob() {
  if (isShuttingDown) return;

  const task = await queue.poll();
  if (!task) return;

  activeJobs++;
  processJob(task).finally(() => {
    activeJobs--;
  });
}

function shutdown() {
  console.log('SIGTERM received, draining...');
  isShuttingDown = true;

  // Stop accepting new work (e.g., stop polling, close consumers)
  consumer.stop();

  // Wait for in-flight work to finish
  const interval = setInterval(() => {
    if (activeJobs === 0) {
      clearInterval(interval);
      console.log('All jobs drained, exiting');
      process.exit(0);
    }
  }, 1000);
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

Use context.Context cancellation or os/signal to handle shutdown:

worker.go

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)

go func() {
    <-sigCh
    log.Println("SIGTERM received, shutting down...")
    cancel()
}()

// Pass ctx to your worker — it should stop accepting new work
// when the context is cancelled and drain in-flight tasks
worker.Run(ctx)

Step 2: Ensure the process receives SIGTERM

A common pitfall is wrapping your application in a shell script or bash -c "..." in your Dockerfile. When this happens, bash is PID 1 and your application is a child process — SIGTERM is sent to bash, which does not forward it to child processes by default.

If your Dockerfile uses CMD ["bash", "-c", "..."], CMD npm start, or even CMD ["npm", "start"], your application process may never receive SIGTERM. Shell wrappers swallow the signal, and npm does not forward signals to its child process. This means your graceful shutdown handler will never run.

There are two ways to fix this:

Use exec form (recommended)
Use tini

Use the exec form of CMD so your application is PID 1 and receives signals directly:

Dockerfile

# Bad — bash swallows SIGTERM
CMD ["bash", "-c", "node dist/worker.js"]

# Good — node is PID 1
CMD ["node", "dist/worker.js"]

If you need to run setup commands before starting your app, use an entrypoint script with exec so your application becomes PID 1:

Dockerfile

COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]

docker-entrypoint.sh

#!/bin/bash
source .env
exec node dist/worker.js

tini is a minimal init process that properly forwards signals to child processes. This is useful when you need a shell wrapper but still want signals forwarded:

Dockerfile

RUN apt-get update && apt-get install -y tini

ENTRYPOINT ["tini", "--"]
CMD ["node", "dist/worker.js"]

Step 3: Set terminationGracePeriodSeconds

By default, Ryvn gives your instance 30 seconds to shut down after receiving SIGTERM. If your workers run tasks that take longer than 30 seconds, you need to increase this value to match your longest expected task duration.

Dashboard
GitOps

Navigate to Environments → select your environment → select the installation → Settings → Values, then set terminationGracePeriodSeconds in your configuration.

Set terminationGracePeriodSeconds in your installation’s config block:

# yaml-language-server: $schema=https://api.ryvn.app/v1/schemas/resources.json
kind: ServiceInstallation
metadata:
  name: my-worker
spec:
  service: worker
  environment: production
  config: |
    terminationGracePeriodSeconds: 300
    replicaCount: 3
    resources:
      requests:
        cpu: 0.5
        memory: 512Mi

Set terminationGracePeriodSeconds to at least the wall-clock time of your longest-running task. For example, if your longest workflow activity takes 5 minutes, set it to 300 or higher.

Autoscaling considerations

If you’re using autoscaling with Temporal or RabbitMQ triggers, the same graceful shutdown pattern applies during scale-down events. When autoscaling reduces the replica count, Ryvn terminates excess instances with SIGTERM — your workers need to drain their in-flight work before exiting. The SIGTERM handler ensures that scale-down events don’t kill active work. Combined with an appropriate terminationGracePeriodSeconds, workers will finish their current tasks even as the installation scales down.

# yaml-language-server: $schema=https://api.ryvn.app/v1/schemas/resources.json
kind: ServiceInstallation
metadata:
  name: temporal-worker
spec:
  service: worker
  environment: production
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 10
    triggers:
      - temporal:
          connection: temporal-prod
          taskQueue: worker-queue
          targetQueueSize: 5
  config: |
    terminationGracePeriodSeconds: 300
    resources:
      requests:
        cpu: 0.5
        memory: 512Mi

Checklist

Before deploying, verify the following:

Signal handler

Your worker code listens for SIGTERM and initiates a clean shutdown — stops accepting new work and drains in-flight tasks.

Process receives signals

Your Dockerfile uses exec form (CMD ["node", "..."]) or tini so that your application process is PID 1 and receives SIGTERM directly.

Grace period is sufficient

terminationGracePeriodSeconds in your Ryvn installation config is set to at least the duration of your longest-running task.

Test locally

Send SIGTERM to your worker process locally (kill -TERM <pid>) and verify it drains cleanly before exiting.

Common issues

Worker is killed immediately on deploy

Your process likely isn’t receiving SIGTERM. Check your Dockerfile — if you’re using bash -c, the signal goes to bash, not your app. Switch to exec form or use tini. See Step 2.

Worker shuts down but tasks are still lost

Your terminationGracePeriodSeconds may be too short. Ryvn force-kills the instance after this period. Increase it to exceed your longest task duration. See Step 3.

Worker hangs and never exits

Your shutdown handler may not be calling process.exit() after draining. For Temporal workers, make sure you call process.exit(0) after worker.run() resolves. For custom workers, ensure your drain-check logic eventually exits.

Multiple workers in the same instance

If your instance runs multiple worker processes, each one needs its own SIGTERM handler. Consider using tini as an init process so signals are forwarded to all children, or structure your code to shut down all workers from a single signal handler.

Workflows

Constraints

Compute

Reference

Graceful Shutdown for Workers

How it works

Step 1: Handle SIGTERM in your worker

Step 2: Ensure the process receives SIGTERM

Step 3: Set terminationGracePeriodSeconds

Autoscaling considerations

Checklist

Common issues

Workflows

Constraints

Compute

Reference

​How it works

​Step 1: Handle SIGTERM in your worker

​Step 2: Ensure the process receives SIGTERM

​Step 3: Set terminationGracePeriodSeconds

​Autoscaling considerations

​Checklist

​Common issues

How it works

Step 1: Handle SIGTERM in your worker

Step 2: Ensure the process receives SIGTERM

Step 3: Set terminationGracePeriodSeconds

Autoscaling considerations

Checklist

Common issues