Concurrency Control in Hatchet Workflows

Hatchet provides powerful concurrency control features to help you manage the execution of your workflows. This is particularly useful when you have workflows that may be triggered frequently or have long-running steps, and you want to limit the number of concurrent executions to prevent overloading your system, ensure fairness, or avoid race conditions.

Why use concurrency control?

There are several reasons why you might want to use concurrency control in your Hatchet workflows:

1. Fairness: When you have multiple clients or users triggering workflows, concurrency control can help ensure fair access to resources. By limiting the number of concurrent runs per client or user, you can prevent a single client from monopolizing the system and ensure that all clients get a fair share of the available resources.

2. Resource management: If your workflow steps are resource-intensive (e.g., they make external API calls or perform heavy computations), running too many instances concurrently can overload your system. By limiting concurrency, you can ensure your system remains stable and responsive.

3. Avoiding race conditions: If your workflow steps modify shared resources, running multiple instances concurrently can lead to race conditions and inconsistent data. Concurrency control helps you avoid these issues by ensuring only a limited number of instances run at a time.

4. Compliance with external service limits: If your workflow steps interact with external services that have rate limits, concurrency control can help you stay within those limits and avoid being throttled or blocked.

5. Spike Protection: When you have workflows that are triggered by external events, such as webhooks or user actions, you may experience spikes in traffic that can overwhelm your system. Concurrency control can help you manage these spikes by limiting the number of concurrent runs and queuing new runs until resources become available.

Configuring concurrency control

To configure concurrency control for a workflow in Hatchet, you can add a concurrency field to your workflow definition. The concurrency field is an object with the following properties:

• name (required): A string identifier for the concurrency limit. This is used to identify the limit in logs and the dashboard.
• maxRuns (optional): The maximum number of concurrent runs allowed for a given concurrency key. If not specified, there is no limit.
• limitStrategy (optional): The strategy to use when the concurrency limit is reached. Can be one of:
  • CANCEL_IN_PROGRESS: Cancel the currently running workflow instances for the same concurrency key to free up slots for the new instance.
  • GROUP_ROUND_ROBIN: Distribute workflow instances across available slots in a round-robin fashion based on the key function.
• key (required): A function that takes the workflow context and returns a string key. This key is used to group runs for the purpose of concurrency limiting. For example, you could use this to limit concurrency on a per-user basis.

Here's an example workflow definition with concurrency control:

export const myWorkflow: Workflow = {
  id: "my-workflow",
  description: "My workflow with concurrency control",
  on: {
    event: "my.event",
  },
  steps: [
    // ...
  ],
  concurrency: {
    name: "my-workflow-concurrency",
    maxRuns: 10,
    limitStrategy: ConcurrencyLimitStrategy.CANCEL_IN_PROGRESS,
    key: (ctx) => ctx.userId,
  },
};

In this example, the workflow is limited to a maximum of 10 concurrent runs for each unique userId in the workflow context. When the limit is reached for a specific userId, new runs with the same userId are queued until a slot becomes available. If the limit strategy is set to CANCEL_IN_PROGRESS, and an event with a conflicting userId is received, the currently running workflow instances for that userId are canceled to free up slots for the new instance.

Setting concurrency on workers

In addition to setting concurrency limits at the workflow level, you can also control concurrency at the worker level by passing the maxRuns option when creating a new Worker instance:

const worker = hatchet.worker("my-worker", {
  maxRuns: 5,
});

This limits the worker to a maximum of 5 concurrent step runs across all workflows. Once the limit is reached, the worker will not accept new actions until a running step completes.

Worker-level concurrency limits are independent of workflow-level limits. The Hatchet engine automatically distributes actions to available workers, and queues actions if all workers are at their concurrency limit.

By combining workflow-level and worker-level concurrency controls, you can fine-tune your Hatchet system for optimal performance and resource utilization.