Concurrency Control in Hatchet Tasks

Hatchet provides powerful concurrency control features to help you manage the execution of your tasks. This is particularly useful when you have tasks that may be triggered frequently or have long-running steps, and you want to limit the number of concurrent executions to prevent overloading your system, ensure fairness, or avoid race conditions. Concurrency strategies can be added to both tasks and workflows

Why use concurrency control?

There are several reasons why you might want to use concurrency control in your Hatchet tasks:

1. Fairness: When you have multiple clients or users triggering tasks, concurrency control can help ensure fair access to resources. By limiting the number of concurrent runs per client or user, you can prevent a single client from monopolizing the system and ensure that all clients get a fair share of the available resources.

2. Resource management: If your tasks are resource-intensive, running too many instances concurrently can overload your system. By limiting concurrency, you can ensure your system remains stable and responsive.

3. Avoiding race conditions: If your tasks modify shared resources, running multiple instances concurrently can lead to race conditions and inconsistent data. Concurrency control helps you avoid these issues by ensuring only a limited number of instances run at a time.

4. Compliance with external service limits: If your tasks interact with external services that have rate limits, concurrency control can help you stay within those limits and avoid being throttled or blocked.

5. Spike Protection: When you have tasks that are triggered by external events, such as webhooks or user actions, you may experience spikes in traffic that can overwhelm your system. Concurrency control can help you manage these spikes by limiting the number of concurrent runs and queuing new runs until resources become available.

Available Strategies:

• Group Round Robin queues incoming task and workflow runs and only dispatches them to workers and triggers them once an available slot is open.
• Cancel In Progress cancels in-progress instances of the task or workflow with matching concurrency keys in order to free up slots for the newly-triggered task or workflow run.
• Cancel Newest cancels any incoming task or workflow runs for a key once the number of runs in a running state for that key has reached a provided limit.

We’re always open to adding more strategies to fit your needs. Join our discord to let us know.

Group Round Robin

When a new task instance is triggered, the Group Round Robin strategy will:

1. Determine the key that the run belongs to based on the CEL expression defined in the task or workflow’s concurrency configuration.
2. Check if there are any available slots for the computed concurrency key based on the maximum number of concurrent runs allowed by the concurrency configuration.
3. If a slot is available, the new task or workflow starts executing immediately.
4. If no slots are available, the new task or workflow is added to a queue for its key.
5. When a running task instance completes and a slot becomes available for a group, the next queued instance for that group (in round-robin order) is dequeued and starts executing.

Group round robin ensures that task instances are processed fairly across different groups, preventing any one group from monopolizing the available resources. It also helps to reduce latency for instances within each group, as they are processed in a round-robin fashion rather than strictly in the order they were triggered. Group round robin is also useful as a global concurrency control for the maximum number of runs of a single task or workflow that you want executing at any given time, regardless of any grouping. You can set a constant concurrency key as the expression, such as '*', to enable this global concurrency control behavior.

Cancel In Progress

When a new task instance is triggered, the Cancel In Progress strategy will:

1. Determine the key that the run belongs to based on the CEL expression defined in the task or workflow’s concurrency configuration.
2. Check if there are any available slots for the computed concurrency key based on the maximum number of concurrent runs allowed by the concurrency configuration.
3. If a slot is available, the new task or workflow starts executing immediately.
4. If no slots are available, a running instance of the task or workflow with the same concurrency key will be cancelled cancelled to free up a slot, and the new instance will start executing immediately.

Cancel In Progress ensures that the most recently triggered runs always take priority over older ones, which is useful when newer inputs supersede older ones and in-progress work becomes stale or irrelevant as soon as a newer run arrives. It’s particularly well suited for user-facing interactions where only the latest input matters (such as chat messages, form submissions, or search-as-you-type), for resource-intensive tasks where it’s more efficient to abandon an old run than wait for it to complete, and for any scenario where you want to prioritize processing the most recent data or events over older ones.

Cancel Newest

When a new task instance is triggered, the Cancel Newest strategy will:

1. Determine the key that the run belongs to based on the CEL expression defined in the task or workflow’s concurrency configuration.
2. Check if there are any available slots for the computed concurrency key based on the maximum number of concurrent runs allowed by the concurrency configuration.
3. If a slot is available, the new task or workflow starts executing immediately.
4. If no slots are available, the newly triggered run is cancelled immediately, allowing the in-progress runs to continue uninterrupted.

Cancel Newest is the inverse of Cancel In Progress: rather than preempting running work in favor of new arrivals, it protects in-progress runs from being disrupted by allowing them to complete before any new work for the same key is started. This is useful when you want to guarantee that long-running task instances finish without interference, when the cost of restarting work outweighs the value of processing newer inputs, and when you want to prevent a single group’s instances from monopolizing the available slots by rejecting excess runs outright instead of queuing them.

Multiple concurrency strategies

You can also combine multiple concurrency strategies to create a more complex concurrency control system. For example, you can use one group key to represent a specific team, and another group to represent a specific resource in that team, giving you more control over the rate at which tasks are executed.