# Hatchet Documentation
> Hatchet is a distributed task queue and workflow engine for modern applications. It provides durable execution, concurrency control, rate limiting, and observability for background tasks and workflows in Python, TypeScript, and Go.
---
# What is Hatchet?
Hatchet is a developer platform that helps engineering teams build and deploy mission-critical AI agents, durable workflows, and background tasks. It supports applications written in Python, Typescript, Go and Ruby, and can be used as a service through [Hatchet Cloud](https://cloud.hatchet.run) or [self-hosting](/self-hosting) (we're [open-source and 100% MIT-licensed](https://github.com/hatchet-dev/hatchet)). Hatchet provides a full platform for queuing, automatic retries, real-time monitoring, alerting, and logging.
Unlike a traditional queuing system, Hatchet is built around the concept of durability. Every task and agent invocation is durably persisted in Hatchet, allowing for debugging, retries and replays, and more complex features like [durable workflows](/v1/durable-execution).
## Using these docs
Every docs page in the user guide uses inline code snippets across all four SDKs which are generated from tested examples:
## Concepts
There are three primary concepts to understand when getting started with Hatchet:
- **[Tasks](/v1/tasks)** — the fundamental unit of work. A task wraps a single function and gives Hatchet everything it needs to schedule, execute, and observe it.
- **[Workers](/v1/workers)** — long-running processes in your infrastructure that pick up and execute tasks.
- **[Durable Workflows](/v1/durable-execution)** — compose multiple tasks into durable pipelines with dependencies, retries, and checkpointing.
All tasks and workflows are **defined as code**, making them easy to version, test, and deploy.
## Use cases
While Hatchet is a general-purpose orchestration platform, it's particularly well-suited for:
- **AI agents** — Hatchet's durability features allow agents to automatically checkpoint their current state and pick up where they left off when faced with unexpected errors. Hatchet's observability features and distributed-first approach are built for debugging long-running agents at scale.
- **Massive parallelization** - Hatchet is built to handle millions of parallel task executions without overloading your workers. Worker-level slot control allows your workers to only accept the amount of work they can handle, while features like [fairness](/v1/concurrency) and [priorities](/v1/priority) are built to help scale massively parallel ingestion.
- **Mission-critical workloads** - everything in Hatchet is durable by default. This means that every task, DAG, event or agent invocation is stored in a durable event log and ready to be replayed at some point in the future.
## Self Hosting
If you plan on self-hosting or have requirements for an on-premise deployment, there are some additional considerations:
- **Minimal Infra Dependencies** - Hatchet is built on top of PostgreSQL and for simple workloads, [it's all you need](/self-hosting/hatchet-lite).
- **Fully Featured Open Source** - Hatchet is 100% MIT licensed, so you can run the same application code against [Hatchet Cloud](https://cloud.hatchet.run) to get started quickly or [self-host](/self-hosting) when you need more control.
## Production Readiness
Hatchet has been battle-tested in production environments, processing billions of tasks per month for scale-ups and enterprises across various industries. Our open source offering is deployed over 10k times per month, while Hatchet Cloud supports hundreds of companies running at scale.
> With Hatchet, we've scaled our indexing workflows effortlessly, reducing failed runs by 50% and doubling our user base in just two weeks!
> — Soohoon, Co-Founder @ Greptile
> Hatchet enables Aevy to process up to 50,000 documents in under an hour through optimized parallel execution, compared to nearly a week with our previous setup.
> — Ymir, CTO @ Aevy
## Ready to get started?
Get started quickly with the **[Hatchet Cloud Quickstart](/v1/quickstart)** or **[self-hosting](/self-hosting)**.
---
# Hatchet Cloud Quickstart
By the end of this guide you'll have a worker running locally that executes a simple task triggered from the CLI.
> **Info:** This guide walks you through getting set up on Hatchet Cloud. If you'd like to
> self-host Hatchet, please see the [self-hosted quickstart](/self-hosting)
> instead.
### Sign up
If you haven't already signed up for Hatchet Cloud, please register [here](https://cloud.hatchet.run).
### Set up your tenant
In Hatchet Cloud, you'll be shown a screen to create your first organization and tenant. A tenant is a logical separation of your environments (e.g. `dev`, `staging`, `production`). Each tenant has its own set of users who can access it.
After creating the tenant, you can simply follow the instructions in the Hatchet Cloud dashboard to set up your first quickstart project and workflow. We have copied the instructions in the following steps.
### Install the Hatchet CLI
#### Native Install (Recommended)
**MacOS, Linux, WSL**
```sh
curl -fsSL https://install.hatchet.run/install.sh | bash
```
#### Homebrew
**MacOS**
```sh
brew install hatchet-dev/hatchet/hatchet --cask
```
### Set your Hatchet profile
You will need to create a Hatchet CLI profile to connect to your Hatchet Cloud tenant. You can do this using the `hatchet profile add` command:
```sh
hatchet profile add
```
Note that the Hatchet Cloud dashboard will provide you with an API token to use when creating your profile.
### Run the quickstart
You can run the Hatchet Cloud quickstart using the `hatchet quickstart` command:
```sh
hatchet quickstart
```
### Run your worker
After setting up the quickstart project, you can run your worker locally by following the instructions printed after the quickstart command. This will involve using the `hatchet worker dev` command:
```sh
hatchet worker dev
```
### Trigger a workflow
Finally, you can trigger your workflow using the `hatchet trigger simple` command:
```sh
hatchet trigger simple
```
### (Optional) Install Hatchet docs MCP and Agent Skills
Get Hatchet documentation directly in your AI coding assistant (Cursor, Claude Code, and more):
```sh copy
hatchet docs install
```
Get agent skills for common CLI operations:
```sh copy
hatchet skills install
```
See the [full setup guide](/v1/using-coding-agents) for manual configuration options.
And that's it! You should now have a Hatchet project set up on Hatchet Cloud with a worker running locally.
## Next Steps
Once you've completed the quickstart, continue to the next section to learn how to [create your first task](/v1/tasks).
---
McpUrl,
CursorDeeplinkButton,
CursorMcpConfig,
ClaudeCodeCommand,
CursorTabLabel,
ClaudeCodeTabLabel,
OtherAgentsTabLabel,
} from "@/components/McpSetup";
import Keywords from "@/components/Keywords";
# Using Coding Agents
Hatchet is designed to work well with AI coding agents. This page covers how to give your agent access to Hatchet documentation and step-by-step skills for common CLI operations.
> **Info:** **Prerequisite:** The `hatchet skills install` and `hatchet docs install`
> commands require the Hatchet CLI. See the [CLI reference](/cli) for
> installation instructions.
## Agent Skills
Agent skills are reference documents that teach AI coding agents how to use the Hatchet CLI — triggering workflows, starting workers, debugging runs, and more.
Run the following command in your project root to install the skill package:
```bash copy
hatchet skills install
```
This creates a `skills/hatchet-cli/` directory with step-by-step reference files and appends a section to your project's `AGENTS.md` (and `CLAUDE.md`) pointing agents to the right file for each task.
**Install to a custom directory:**
```bash copy
hatchet skills install --dir ./my-project
```
After installation, commit the `skills/` directory and `AGENTS.md` to version control so all agents working in the repo benefit automatically.
### Available references
Reference, When to use
`references/setup-cli.md`, Installing the CLI, creating or listing profiles
`references/start-worker.md`, Starting a dev worker for local development
`references/trigger-and-watch.md`, Triggering a workflow and polling for completion
`references/debug-run.md`, Diagnosing a failed, stuck, or unexpected run
`references/replay-run.md`, Re-running a previous workflow with same or new input
## MCP Server
Hatchet documentation is available as an **MCP (Model Context Protocol) server**, so AI coding assistants like Cursor and Claude Code can search and reference Hatchet docs directly.
MCP endpoint:
#### Hatchet CLI
```bash copy
hatchet docs install claude-code
```
If `claude` is on your PATH, this runs the command automatically. Otherwise it prints it for you to copy.
#### Command
Run this command in your terminal:
For any AI tool that supports [llms.txt](https://llmstxt.org/), Hatchet docs are available at:
| Resource | URL |
|----------|-----|
| **llms.txt** (index) | [docs.hatchet.run/llms.txt](https://docs.hatchet.run/llms.txt) |
| **llms-full.txt** (all docs) | [docs.hatchet.run/llms-full.txt](https://docs.hatchet.run/llms-full.txt) |
| **Per-page markdown** | `docs.hatchet.run/llms/{section}/{page}.md` |
| **MCP endpoint** | |
Every documentation page also includes a `` header
pointing to its markdown version, and a "View as Markdown" link at the top of the page.
## llms.txt
For any AI tool that supports [llms.txt](https://llmstxt.org/), Hatchet docs are available at:
Resource, URL
**llms.txt** (index), [docs.hatchet.run/llms.txt](https://docs.hatchet.run/llms.txt)
**llms-full.txt** (all docs), [docs.hatchet.run/llms-full.txt](https://docs.hatchet.run/llms-full.txt)
**Per-page markdown**, `docs.hatchet.run/llms/{section}/{page}.md`
**MCP endpoint**,
Every documentation page also includes a `` header
pointing to its markdown version, and a "View as Markdown" link at the top of the page.
---
# Tasks
The fundamental unit of work in Hatchet is a **task**. At its most basic level, a task is just a function. You can invoke a task on its own (a "standalone" task), compose tasks into a [DAG workflow](/v1/directed-acyclic-graphs), or use [durable task composition](/v1/child-spawning) to spawn child tasks at runtime.
Every task you invoke is **durable** - Hatchet persists it, its state, and its results even after it finishes running.
## Defining a task
A task needs a name and a function. The function accepts an [input](#input-and-output) and a [context](#the-context-object).
#### Python
```python
class SimpleInput(BaseModel):
message: str
class SimpleOutput(BaseModel):
transformed_message: str
# Declare the task to run
@hatchet.task(name="first-task", input_validator=SimpleInput)
def first_task(input: SimpleInput, ctx: Context) -> SimpleOutput:
print("first-task task called")
return SimpleOutput(transformed_message=input.message.lower())
```
In the Python SDK, the arguments to the task are passed _positionally_, which means you can name them whatever you like. For instance, defining a task as `async def my_task(foo: Input, bar: Context) -> None:` is perfectly valid. Tasks in Hatchet can be either sync or async, although we generally recommend trying to use async wherever possible.
#### Typescript
```typescript
import { hatchet } from '../hatchet-client';
// (optional) Define the input type for the workflow
export type SimpleInput = {
Message: string;
};
export const simple = hatchet.task({
name: 'simple',
retries: 3,
fn: async (input: SimpleInput) => {
return {
TransformedMessage: input.Message.toLowerCase(),
};
},
});
```
In the TypeScript SDK, the `fn` argument is a function that takes the workflow's input and a context object. The context object contains information about the workflow run (e.g. the run ID, the workflow's input, etc). It can be synchronous or asynchronous.
#### Go
```go
type SimpleInput struct {
Message string `json:"message"`
}
type SimpleOutput struct {
Result string `json:"result"`
}
task := client.NewStandaloneTask("process-message", func(ctx hatchet.Context, input SimpleInput) (SimpleOutput, error) {
return SimpleOutput{
Result: "Processed: " + input.Message,
}, nil
})
```
#### Ruby
```ruby
FIRST_TASK = HATCHET.task(name: "first-task") do |input, ctx|
puts "first-task called"
{ "transformed_message" => input["message"].downcase }
end
```
## Input and output
Every task receives an **input** - a JSON-serializable object passed when the task is triggered. The value that is returned from the task becomes the task's **output**, which callers receive when they await the result of the task.
Hatchet's SDKs support type-checked and runtime-validated input and output types for tasks, so that you can integrate your Hatchet tasks into your codebase in a type-safe and predictable way that provides you all of the guarantees you get from, for example, replacing the Hatchet task run with a local function call.
You can refer to the [examples above](#defining-a-task) to see how to provide validators for task inputs and outputs.
## The context object
In addition to input and output payloads, every task receives a **context**. The context provides Hatchet-related information that might be useful to the execution of the task at runtime. For instance, you might access the workflow run ID, the task run ID, or the retry count from the context and have your task's application logic do something with those values.
The context also provides helper methods for interacting with a number of Hatchet's features, such as [managing cancellations](/v1/cancellation), [refreshing timeouts](/v1/timeouts#refreshing-timeouts), [publishing stream events](/v1/streaming#publishing-stream-events) and more.
#### Python
See the [Python SDK reference](/reference/python/context) for more details
#### Typescript
See the [TypeScript SDK reference](/reference/typescript/Context) for more
details
#### Go
See the [Go SDK
reference](https://pkg.go.dev/github.com/hatchet-dev/hatchet/sdks/go#Context)
for more details
#### Ruby
Ruby SDK reference coming soon! For now, see the [Python SDK
reference](/reference/python/context) to get a sense of what's available.
## Configuration
Tasks can be configured to handle common problems in distributed systems. For example, you might want to automatically retry a task when an external API returns a transient error, or limit how many instances of a task run at the same time to avoid overwhelming a downstream service.
Concept, What it does
[Retries](/v1/retry-policies), Retry the task on failure, with optional backoff.
[Timeouts](/v1/timeouts), Limit how long a task may wait to be scheduled or to run.
[Concurrency](/v1/concurrency), Distribute load fairly between your customers.
[Rate limits](/v1/rate-limits), Throttle task execution over a time window.
[Priority](/v1/priority), Influence scheduling order relative to other queued tasks.
[Worker affinity](/v1/advanced-assignment/worker-affinity), Prefer or require specific workers for this task.
## How tasks execute on workers
Tasks don't run on their own. [Workers](/v1/workers) execute them. A worker is a long-running process that registers one or more tasks with Hatchet. When you trigger a task, Hatchet places it in a queue and assigns it to an available worker that has registered that task. When a task completes, the Hatchet SDK running on the worker sends the result back to Hatchet, which marks the task as a success or failure, displays the results, and so on.
---
# Workers
Workers in Hatchet are the long-running processes that execute [tasks](/v1/tasks). In the broadest sense, it may be helpful to think of a worker as a simple `while` loop that receives a new task assignment from Hatchet, executes the task, and reports the results back.
When workers are spun up - in any environment, be it locally, on a VM, etc. - they will register themselves with Hatchet to start receiving and executing tasks.
## Declaring a worker
A worker needs a name and a set of tasks (or workflows, more on this later) to register:
#### Python
```python
def main() -> None:
worker = hatchet.worker("dag-worker", workflows=[dag_workflow])
worker.start()
```
#### Typescript
```typescript
import { hatchet } from '../hatchet-client';
import { simple } from './workflow';
import { parent, child } from './workflow-with-child';
import { simpleWithZod } from './zod';
async function main() {
const worker = await hatchet.worker('simple-worker', {
// 👀 Declare the workflows that the worker can execute
workflows: [simple, simpleWithZod, parent, child],
// 👀 Declare the number of concurrent task runs the worker can accept
slots: 100,
});
await worker.start();
}
if (require.main === module) {
main();
}
```
#### Go
```go
worker, err := client.NewWorker("simple-worker", hatchet.WithWorkflows(task))
if err != nil {
log.Fatalf("failed to create worker: %v", err)
}
interruptCtx, cancel := cmdutils.NewInterruptContext()
defer cancel()
err = worker.StartBlocking(interruptCtx)
if err != nil {
log.Fatalf("failed to start worker: %v", err)
}
```
#### Ruby
```ruby
def main
worker = HATCHET.worker("dag-worker", workflows: [DAG_WORKFLOW])
worker.start
end
```
When a worker starts, it registers each of its tasks and workflows with Hatchet. From that point on, Hatchet knows to route matching tasks to that worker.
One important note is that multiple workers can register the same task. In this scenario, Hatchet distributes work across all of them, allowing for simple horizontal scaling.
## Starting a worker
#### CLI (recommended)
The fastest way to run a worker during development is with the Hatchet CLI. This handles authentication and hot reloads on code changes:
```bash
hatchet worker dev
```
#### Script
You can also run workers without the CLI, which you're likely to do in a production setting, for instance. To do this, you'll first need to set a `HATCHET_CLIENT_TOKEN` environment variable, or provide it via parameters when creating the Hatchet client.
> **Info:** If you don't already have a token, you can generate one in the "API Tokens" section under "Settings" in the dashboard.
```bash
export HATCHET_CLIENT_TOKEN=""
```
If you're running a self-hosted engine without TLS enabled, also set:
```bash
export HATCHET_CLIENT_TLS_STRATEGY=none
```
Then run the worker:
#### Python
```bash
python worker.py
```
#### Typescript
Add a script to your `package.json`:
```json
"scripts": {
"start:worker": "ts-node src/worker.ts"
}
```
Then run it:
```bash
npm run start:worker
```
#### Go
```bash
go run main.go
```
#### Ruby
```bash
bundle exec ruby worker.rb
```
Once the worker starts, you will see logs confirming it is connected:
```
[INFO] 🪓 -- starting hatchet...
[DEBUG] 🪓 -- 'test-worker' waiting for ['simpletask:step1']
[DEBUG] 🪓 -- acquired action listener: efc4aaf2-...
[DEBUG] 🪓 -- sending heartbeat
```
> **Info:** For self-hosted engines, there may be additional gRPC configuration options
> needed. See the [Self-Hosting](/self-hosting/worker-configuration-options)
> docs for details.
## Slots
Every worker has a fixed number of **slots** that control how many tasks it can run concurrently, which can be configured with the `slots` option on the worker. For instance, if `slots` is set to 5, the worker will run up to five tasks concurrently at any time. Any additional tasks wait in the queue until a slot opens up. Slots are a **local** limit. They protect the individual worker from attempting to run more tasks concurrently than desired, which can help control resource usage by the worker.
To set the slot count, pass the `slots` option when declaring the worker:
#### Python
```python
def main() -> None:
worker = hatchet.worker(
"concurrency-demo-worker", slots=10, workflows=[concurrency_limit_workflow]
)
worker.start()
```
#### Typescript
```typescript
async function main() {
const worker = await hatchet.worker('timeout-worker', {
workflows: [timeoutTask, refreshTimeoutTask],
slots: 50,
});
await worker.start();
}
```
#### Go
```go
worker, err := client.NewWorker("concurrency-worker",
hatchet.WithWorkflows(
ConcurrencyRoundRobin(client),
MultipleConcurrencyKeys(client),
ConcurrencyCancelInProgress(client),
ConcurrencyCancelNewest(client),
),
hatchet.WithSlots(10),
)
if err != nil {
log.Fatalf("failed to create worker: %v", err)
}
```
#### Ruby
```ruby
def main
worker = HATCHET.worker(
"concurrency-demo-worker", slots: 10, workflows: [CONCURRENCY_LIMIT_WORKFLOW]
)
worker.start
end
```
The default slot count for workers in Hatchet is 100. In many cases, leaving the default as-is will be perfectly fine, especially when first getting set up with Hatchet.
By default each running task consumes one slot. A task can be configured to consume more than one, so a task that needs more memory or CPU takes up more of a worker's capacity. See [Task Slot Cost](/v1/advanced-assignment/slot-cost).
---
# Running Tasks
Once you've [defined some tasks](/v1/tasks) and registered them on a [worker](/v1/workers), you're ready to run them!
Hatchet lets you run tasks in a number of ways, which support different application needs. In broad strokes, these are [fire-and-wait](#fire-and-wait), various forms of [fire-and-forget-style triggering](#fire-and-forget), and scheduling tasks to run either [periodically](#crons) or at some [specific time in the future](#scheduled-runs).
## Fire-and-wait
Fire-and-wait is a common way of triggering a task which blocks until it completes and returns a result. This is particularly useful for situations where you want to do something with the result of your task. For instance, if your task generates some LLM output which you want to return to the user or persist in the database, you might trigger a task, wait for it to complete, collect its result, and then continue on with your application logic.
#### Python
Call `run` or `aio_run` on a `Task` or `Workflow` object to invoke it. These methods block until the task or workflow completes and return the result.
```python
from examples.child.worker import SimpleInput, child_task
child_task.run(SimpleInput(message="Hello, World!"))
```
The run methods in Hatchet's Python SDK also have async flavors you can `await`. These are prefixed with `aio_`, such as `aio_run`.
```python
result = await child_task.aio_run(SimpleInput(message="Hello, World!"))
```
Note that the type of `input` here is a Pydantic model that matches the input schema of the task or workflow being triggered.
#### Typescript
Call `run` on the `Task` object to invoke it. This returns a promise that resolves when the task completes and returns the result.
```typescript
const res = await parent.run(
{
Message: 'HeLlO WoRlD',
},
{
additionalMetadata: {
test: 'test',
},
}
);
const res3 = await simpleWithZod.run({
Message: 'HeLlO WoRlD',
});
console.log(res3.TransformedMessage);
// 👀 Access the results of the Task
console.log(res.TransformedMessage);
```
#### Go
Call `Run` on the `Task` object to invoke it. This blocks until the task completes and returns the result.
```go
result, err := task.Run(context.Background(), SimpleInput{Message: "Hello, World!"})
if err != nil {
return err
}
```
#### Ruby
Call `run` on the `Task` object to invoke it. This blocks until the task completes and returns the result.
```ruby
result = CHILD_TASK_WF.run({ "message" => "Hello, World!" })
```
## Fire-and-forget
On the other hand, fire-and-forget-style triggering enqueues a task without waiting for the result. This is useful for background jobs like sending emails, processing uploads, or kicking off long-running pipelines where the application does _not_ need to wait for the result to continue along.
#### Python
Call `run` with `wait_for_result=False` on a `Task` or `Workflow` object to enqueue it fire-and-forget. This returns a `WorkflowRunRef` you can use to access the run ID and result later.
```python
ref = say_hello.run(input=HelloInput(name="World"), wait_for_result=False)
```
There's also an async flavor:
```python
ref = await say_hello.aio_run(
input=HelloInput(name="Async World"), wait_for_result=False
)
```
Note that the type of `input` here is a Pydantic model that matches the input schema of the task.
#### Typescript
Call `runNoWait` on the `Task` object to enqueue it without waiting for the result. This returns a `WorkflowRunRef`.
```typescript
import { simple } from './workflow';
// ...
async function main() {
// 👀 Enqueue the workflow
const run = await simple.runNoWait({
Message: 'hello',
});
// 👀 Get the run ID of the workflow
const runId = await run.getWorkflowRunId();
// It may be helpful to store the run ID of the workflow
// in a database or other persistent storage for later use
console.log(runId);
```
#### Go
Call `RunNoWait` on the `Task` object to enqueue it without waiting for the result. This returns a `WorkflowRunRef`.
```go
runRef, err := task.RunNoWait(context.Background(), SimpleInput{Message: "Hello, World!"})
if err != nil {
return err
}
fmt.Println(runRef.RunId)
```
#### Ruby
Call `run_no_wait` on the `Task` object to enqueue it without waiting for the result. This returns a `WorkflowRunRef`.
```ruby
ref = SAY_HELLO.run_no_wait({ "name" => "World" })
```
When running a task fire-and-forget-style, you can also always retrieve the result later on. The workflow run ref that's returned from these trigger methods has a result method, which lets you retrieve the result of the triggered task if you need it later.
#### Python
Use `ref.result()` to block until the result is available:
```python
result = ref.result()
```
or await `aio_result`:
```python
result = await ref.aio_result()
```
#### Typescript
```typescript
// the return object of the enqueue method is a WorkflowRunRef which includes a listener for the result of the workflow
const result = await run.output;
console.log(result);
// if you need to subscribe to the result of the workflow at a later time, you can use the runRef method and the stored runId
const ref = hatchet.runRef(runId);
const result2 = await ref.output;
console.log(result2);
```
#### Go
```go
result, err := runRef.Result()
if err != nil {
return err
}
var resultOutput SimpleOutput
err = result.TaskOutput("process-message").Into(&resultOutput)
if err != nil {
return err
}
fmt.Println(resultOutput.Result)
```
#### Ruby
```ruby
result = ref.result
```
### Triggering from events
Another method of running tasks fire-and-forget style is via [pushing events](/v1/events#pushing-events-to-hatchet) to Hatchet. If a task is configured to be triggered by an event, then when the event is pushed to Hatchet, a corresponding run will be triggered using the payload of the event as the input to the run.
You can push events to Hatchet directly using the SDKs, or via [webhooks](/v1/webhooks), which are converted into Hatchet events internally on ingestion.
## Crons
Another common way to run tasks is on a cron schedule, which Hatchet [supports natively](/v1/cron-runs). Crons are useful for running tasks that are expected to run at the same time every day, such as data processing pipelines, reconciliation jobs, and so on.
Note that Hatchet supports second-level cron granularity, although in most cases using the minutes as the most granular level is perfectly fine.
## Scheduled runs
Finally, Hatchet also supports [scheduling a run to be triggered at a specific time in the future](/v1/scheduled-runs). This is particularly useful for situations where you want to wait a known amount of time before running some task, such as sending a follow up email or a welcome email to a new user, or allowing your customers to choose when they want something to run, such as sending a reminder, for instance.
## Triggering from the dashboard
Finally, there are a number of pages in the dashboard that have a `Trigger Run` button. You can provide run parameters such as input, additional metadata, and a scheduled time.
---
# Introduction to Durable Execution
At its core, Hatchet is a _**durable execution**_ platform. Unfortunately, durable execution is an overloaded, often-confusing term. If you're new to durable execution, or are curious for a refresher, we wrote a [blog post outlining the core ideas](https://hatchet.run/blog/durable-execution).
At its most basic level, durable execution provides a toolbox that, when used correctly, gives you some guarantees about tasks and workflows you write in Hatchet that you wouldn't get from an ordinary task queueing system.
## Guarantees
One of the main promises of durable execution, when used correctly, is to give your tasks something closer to [exactly-once semantics](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) than you'd get from traditional task queues. In practice, this means that a durable task can guarantee that your application logic is cached correctly and retry-safe, such that every time a piece of a durable task completes, it creates a new checkpoint (an entry in a durable event log), from which we can replay without needing to re-execute the actual application logic.
This means that if you run a durable task to a midway point and the worker it's running on crashes, you can replay the task from whatever checkpoint it last reached without re-running any of the previous steps or duplicating any work. This is priceless in systems that cannot reasonably be made idempotent, so replaying on failure is impossible.
## Core Assumptions
The core assumption of durable execution in Hatchet is that durable tasks only do one of two things: They can _wait_ for something, such as a [sleep to complete](/v1/durable-sleep) or an [event to be received](/v1/durable-event-waits), or they can _[spawn child tasks](/v1/child-spawning)_. These operations can also be composed, such that you can have a durable task wait for _either_ a sleep to complete _or_ an event to be pushed, whichever comes first. You can achieve this behavior by using [or groups](/v1/directed-acyclic-graphs#waiting-on-conditions-with-or-groups).
## Example Uses
There are lots of cases where durable execution is useful. A few common ones where it's an obvious choice are:
1. Agentic workflows, especially ones that require human-in-the-loop steps, which continuously spawn children, collect results, spawn more children, and so on, in a loop. Durable tasks are an obvious fit here, since the durable task can be replayed from where it left off without losing any of the progress that was made by the agent in the past, and without needing to e.g. replay the human-in-the-loop portions of the task, such as approvals or similar.
2. Tasks that are hard to make idempotent, where we cannot replay part of the task once it's completed. For example, something that involves sending an email to a customer, or updating a value in a table midway through.
3. Dynamic workflows, where we build a DAG at runtime by selecting which child workflows to spawn based on the input to the durable task or the results of upstream checkpoints. This is particularly useful for powering tools like drag-and-drop DAG builders.
## Learn More!
There are lots of durable execution concepts and features to cover, and we're only just scratching the surface here! Check out our more detailed documentation on [durable tasks](/v1/durable-tasks), [durable sleeps](/v1/durable-sleep), [durable event waits](/v1/durable-event-waits), and [DAGs](/v1/directed-acyclic-graphs) to keep learning and building.
---
# Scheduled Runs
Scheduled runs allow you to trigger a task at a specific time in the future. Some example use cases of scheduling runs might include:
- Sending a reminder email at a specific time after a user took an action.
- Running a one-time maintenance task at a predetermined time as determined by your application. For instance, you might want to run a database vacuum during a maintenance window any time a task matches a certain criteria.
- Allowing a customer to decide when they want your application to perform a specific task. For instance, if your application is a simple alarm app that sends a customer a notification at a time that they specify, you might create a scheduled run for each alarm that the customer sets.
Hatchet supports scheduled runs to run on a schedule defined in a few different ways:
- [Programmatically](/v1/scheduled-runs#programmatically-creating-scheduled-runs): Use the Hatchet SDKs to dynamically set the schedule of a task.
- [Hatchet Dashboard](/v1/scheduled-runs#managing-scheduled-runs-in-the-hatchet-dashboard): Manually create scheduled runs from the Hatchet Dashboard.
> **Warning:** The scheduled time is when Hatchet **enqueues** the task, not when the run
> starts. Scheduling constraints like concurrency limits, rate limits, and retry
> policies can affect run start times.
## Programmatically Creating Scheduled Runs
### Create a Scheduled Run
You can create dynamic scheduled runs programmatically via the API to run tasks at a specific time in the future.
Here's an example of creating a scheduled run to trigger a task tomorrow at noon:
#### Python
```python
from datetime import datetime
from examples.simple.worker import simple
schedule = simple.schedule(datetime(2025, 3, 14, 15, 9, 26))
## 👀 do something with the id
print(schedule.id)
```
#### Typescript
```typescript
const runAt = new Date(new Date().setHours(12, 0, 0, 0) + 24 * 60 * 60 * 1000);
const scheduled = await simple.schedule(runAt, {
Message: 'hello',
});
// 👀 Get the scheduled run ID of the workflow
// it may be helpful to store the scheduled run ID of the workflow
// in a database or other persistent storage for later use
const scheduledRunId = scheduled.metadata.id;
console.log(scheduledRunId);
```
#### Go
```go
scheduledRun, err := client.Schedules().Create(
context.Background(),
"scheduled",
features.CreateScheduledRunTrigger{
TriggerAt: time.Now().Add(1 * time.Minute),
Input: map[string]interface{}{"message": "Hello, World!"},
},
)
if err != nil {
log.Fatalf("failed to create scheduled run: %v", err)
}
```
#### Ruby
```ruby
schedule = SIMPLE.schedule(Time.now + 86_400, input: { "message" => "Hello, World!" })
## do something with the id
puts schedule.metadata.id
```
In this example you can have different scheduled times for different customers, or dynamically set the scheduled time based on some other business logic.
When creating a scheduled run via the API, you will receive a scheduled run object with a metadata property containing the id of the scheduled run. This id can be used to reference the scheduled run when deleting the scheduled run and is often stored in a database or other persistence layer.
> **Info:** Note: Be mindful of the time zone of the scheduled run. Scheduled runs are
> **always** stored and returned in UTC.
### Deleting a Scheduled Run
You can delete a scheduled run by calling the `delete` method on the scheduled client.
#### Python
```python
hatchet.scheduled.delete(scheduled_id=scheduled_run.metadata.id)
```
#### Typescript
```typescript
await hatchet.scheduled.delete(scheduled);
```
#### Go
```go
err = client.Schedules().Delete(
context.Background(),
scheduledRun.Metadata.Id,
)
if err != nil {
log.Fatalf("failed to delete scheduled run: %v", err)
}
```
#### Ruby
```ruby
hatchet.scheduled.delete(scheduled_run.metadata.id)
```
### Listing Scheduled Runs
You can list all scheduled runs for a task by calling the `list` method on the scheduled client.
#### Python
```python
scheduled_runs = hatchet.scheduled.list()
```
#### Typescript
```typescript
const scheduledRuns = await hatchet.scheduled.list({
workflow: simple,
});
console.log(scheduledRuns);
```
#### Go
```go
scheduledRuns, err := client.Schedules().List(
context.Background(),
rest.WorkflowScheduledListParams{},
)
if err != nil {
log.Fatalf("failed to list scheduled runs: %v", err)
}
```
#### Ruby
```ruby
scheduled_runs = hatchet.scheduled.list
```
### Rescheduling a Scheduled Run
If you need to change the trigger time for an existing scheduled run, you can reschedule it by updating its `triggerAt`.
#### Python
```python
hatchet.scheduled.update(
scheduled_id=scheduled_run.metadata.id,
trigger_at=datetime.now(tz=timezone.utc) + timedelta(hours=1),
)
```
#### Typescript
```typescript
await hatchet.scheduled.update(scheduledRunId, {
triggerAt: new Date(Date.now() + 60 * 60 * 1000),
});
```
#### Ruby
```ruby
hatchet.scheduled.update(
scheduled_run.metadata.id,
trigger_at: Time.now + 3600
)
```
> **Warning:** You can only reschedule scheduled runs created via the API (not runs created
> via a code-defined schedule), and Hatchet may reject rescheduling if the run
> has already triggered.
### Bulk operations (delete / reschedule)
Hatchet supports bulk operations for scheduled runs. You can bulk delete scheduled runs, and you can bulk reschedule scheduled runs by providing a list of updates.
#### Python
```python
hatchet.scheduled.bulk_delete(scheduled_ids=[id])
hatchet.scheduled.bulk_delete(
workflow_id="workflow_id",
statuses=[ScheduledRunStatus.SCHEDULED],
additional_metadata={"customer_id": "customer-a"},
)
```
```python
hatchet.scheduled.bulk_update(
[
(id, datetime.now(tz=timezone.utc) + timedelta(hours=2)),
]
)
```
#### Typescript
```typescript
await hatchet.scheduled.bulkDelete({
scheduledRuns: [scheduledRunId],
});
```
```typescript
await hatchet.scheduled.bulkUpdate([
{ scheduledRun: scheduledRunId, triggerAt: new Date(Date.now() + 2 * 60 * 60 * 1000) },
]);
```
#### Ruby
```ruby
hatchet.scheduled.bulk_delete(scheduled_ids: [id])
```
```ruby
hatchet.scheduled.bulk_update(
[[id, Time.now + 7200]]
)
```
## Managing Scheduled Runs in the Hatchet Dashboard
In the Hatchet Dashboard, you can view and manage scheduled runs for your tasks.
Navigate to "Triggers" > "Scheduled Runs" in the left sidebar and click "Create Scheduled Run" at the top right.
You can specify run parameters such as Input, Additional Metadata, and the Scheduled Time.
You can also manage existing scheduled runs:
- **Single-run actions**: Use the per-row actions menu to **Reschedule** or **Delete** an individual scheduled run.
- **Bulk actions**: Use the **Actions** menu to bulk **Delete** or **Reschedule** either:
- The selected rows, or
- All rows matching the current filters (including “all” if no filters are set).
> **Info:** In the dashboard, reschedule/delete actions may be disabled for runs that were
> created via a code-defined schedule, and rescheduling may be disabled for runs
> that have already triggered.
## Scheduled Run Considerations
When using scheduled runs, there are a few considerations to keep in mind:
1. **Time Zone**: Scheduled runs are stored and returned in UTC. Make sure to consider the time zone when defining your scheduled time.
2. **Execution Time**: The actual execution time of a scheduled run may vary slightly from the scheduled time. Hatchet makes a best-effort attempt to enqueue the task as close to the scheduled time as possible, but there may be slight delays due to system load or other factors.
3. **Missed Schedules**: If a scheduled task is missed (e.g., due to system downtime), Hatchet will not automatically run the missed instances when the service comes back online.
4. **Overlapping Schedules**: If a task is still running when a second scheduled run is scheduled to start, Hatchet will start a new instance of the task or respect [concurrency](/v1/concurrency) policy.
---
# Recurring Runs with Cron
A [Cron](https://en.wikipedia.org/wiki/Cron) is a time-based job scheduler that allows you to define when a task should be executed automatically on a pre-determined schedule.
Some example use cases for cron-style tasks might include:
1. Running a daily report at a specific time.
2. Sending weekly digest emails to users about their activity from the past week.
3. Running a monthly billing process to generate invoices for customers.
Hatchet supports cron triggers to run on a schedule defined in a few different ways:
- [Task Definitions](/v1/cron-runs#defining-a-cron-in-your-task-definition): Define a cron expression in your task definition to trigger the task on a predefined schedule.
- [Dynamic Programmatically](/v1/cron-runs#programmatically-creating-cron-triggers): Use the Hatchet SDKs to dynamically set the cron schedule of a task.
- [Hatchet Dashboard](/v1/cron-runs#managing-cron-jobs-in-the-hatchet-dashboard): Manually create cron triggers from the Hatchet Dashboard.
> **Warning:** The expression is when Hatchet **enqueues** the task, not when the run starts.
> Scheduling constraints like concurrency limits, rate limits, and retry
> policies can affect run start times.
### Cron Expression Syntax
Cron expressions in Hatchet follow the standard cron syntax. Hatchet supports both 5-field and 6-field expressions.
A cron expression consists of 5 to 6 fields separated by spaces. If there are 6 fields, the first field is seconds; if
there are 5 fields, the first field is minutes.
```
┌───────────── second (0 - 59) (optional)
│ ┌───────────── minute (0 - 59)
│ │ ┌───────────── hour (0 - 23)
│ │ │ ┌───────────── day of the month (1 - 31)
│ │ │ │ ┌───────────── month (1 - 12)
│ │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
* * * * * *
```
Each field can contain a specific value, an asterisk (`*`) to represent all possible values, or a range of values. Here are some examples of cron expressions:
- `0 0 * * *`: Run every day at midnight
- `*/15 * * * *`: Run every 15 minutes
- `0 9 * * 1`: Run every Monday at 9 AM
- `0 0 1 * *`: Run on the first day of every month at midnight
- `30 * * * * *`: Run at 30 seconds past every minute (6-field)
> **Info:** Keep in mind, Hatchet Cloud meters by Task runs so use seconds wisely. Have
> questions about pricing? [Contact us](https://cal.com/team/hatchet/talk-to-us)
## Defining a Cron in Your Task Definition
You can define a task with a cron schedule by configuring the cron expression as part of the task definition:
#### Python-Sync
```python
# Adding a cron trigger to a workflow is as simple
# as adding a `cron expression` to the `on_cron`
# prop of the workflow definition
cron_workflow = hatchet.workflow(name="CronWorkflow", on_crons=["* * * * *"])
@cron_workflow.task()
def step1(input: EmptyModel, ctx: Context) -> dict[str, str]:
return {
"time": "step1",
}
```
#### Python-Async
```python
# Adding a cron trigger to a workflow is as simple
# as adding a `cron expression` to the `on_cron`
# prop of the workflow definition
cron_workflow = hatchet.workflow(name="CronWorkflow", on_crons=["* * * * *"])
@cron_workflow.task()
def step1(input: EmptyModel, ctx: Context) -> dict[str, str]:
return {
"time": "step1",
}
```
#### Typescript
```typescript
export const onCron = hatchet.workflow({
name: 'on-cron-workflow',
on: {
// 👀 add a cron expression to run the workflow every 15 minutes
cron: '*/15 * * * *',
},
});
```
#### Go
```go
dailyCleanup := client.NewStandaloneTask("cleanup-temp-files", func(ctx hatchet.Context, input CronInput) (CronOutput, error) {
log.Printf("Running daily cleanup at %s", input.Timestamp)
time.Sleep(2 * time.Second)
return CronOutput{
JobName: "daily-cleanup",
ExecutedAt: time.Now().Format(time.RFC3339),
NextRun: "Next run: tomorrow at 2 AM",
}, nil
},
hatchet.WithWorkflowCron("0 2 * * *"),
hatchet.WithWorkflowCronInput(CronInput{
Timestamp: time.Now().Format(time.RFC3339),
}),
hatchet.WithWorkflowDescription("Daily cleanup and maintenance tasks"),
)
```
#### Ruby
```ruby
CRON_WORKFLOW = HATCHET.workflow(
name: "CronWorkflow",
on_crons: ["*/5 * * * *"]
)
CRON_WORKFLOW.task(:cron_task) do |input, ctx|
puts "Cron task executed at #{Time.now}"
{ "status" => "success" }
end
```
In the examples above, we set the `on cron` property of the task. The property specifies the cron expression that determines when the task should be triggered.
Note: When modifying a cron in your task definition, it will override any cron
schedule for previous crons defined in previous task definitions, but crons
created via the API or Dashboard will still be respected.
## Programmatically Creating Cron Triggers
### Create a Cron Trigger
You can create dynamic cron triggers programmatically via the API. This is useful if you want to create a cron trigger that is not known at the time of task definition,
Here's an example of creating a a cron to trigger a report for a specific customer every day at noon:
#### Python-Sync
```python
cron_trigger = dynamic_cron_workflow.create_cron(
cron_name="customer-a-daily-report",
expression="0 12 * * *",
input=DynamicCronInput(name="John Doe"),
additional_metadata={
"customer_id": "customer-a",
},
)
id = cron_trigger.metadata.id # the id of the cron trigger
```
#### Python-Async
```python
cron_trigger = await dynamic_cron_workflow.aio_create_cron(
cron_name="customer-a-daily-report",
expression="0 12 * * *",
input=DynamicCronInput(name="John Doe"),
additional_metadata={
"customer_id": "customer-a",
},
)
cron_trigger.metadata.id # the id of the cron trigger
```
#### Typescript
```typescript
const cron = await simple.cron('simple-daily', '0 0 * * *', {
Message: 'hello',
});
// it may be useful to save the cron id for later
const cronId = cron.metadata.id;
```
#### Go
```go
createdCron, err := client.Crons().Create(context.Background(), "cleanup-temp-files", features.CreateCronTrigger{
Name: "daily-cleanup",
Expression: "0 0 * * *",
Input: map[string]interface{}{
"timestamp": time.Now().Format(time.RFC3339),
},
AdditionalMetadata: map[string]interface{}{
"description": "Daily cleanup and maintenance tasks",
},
})
if err != nil {
return err
}
```
#### Ruby
```ruby
cron_trigger = dynamic_cron_workflow.create_cron(
"customer-a-daily-report",
"0 12 * * *",
input: { "name" => "John Doe" }
)
id = cron_trigger.metadata.id
```
In this example you can have different expressions for different customers, or dynamically set the expression based on some other business logic.
When creating a cron via the API, you will receive a cron trigger object with a metadata property containing the id of the cron trigger. This id can be used to reference the cron trigger when deleting the cron trigger and is often stored in a database or other persistence layer.
Note: Cron Name and Expression are required fields when creating a cron
trigger and we enforce a unique constraint on the two.
### Delete a Cron Trigger
You can delete a cron trigger by passing the cron object or a cron trigger id to the delete method.
#### Python-Sync
```python
hatchet.cron.delete(cron_id=cron_trigger.metadata.id)
```
#### Python-Async
```python
await hatchet.cron.aio_delete(cron_id=cron_trigger.metadata.id)
```
#### Typescript
```typescript
await hatchet.crons.delete(cronId);
```
#### Go
```go
err = client.Crons().Delete(context.Background(), createdCron.Metadata.Id)
if err != nil {
return err
}
```
#### Ruby
```ruby
hatchet.cron.delete(cron_trigger.metadata.id)
```
Note: Deleting a cron trigger will not cancel any currently running instances
of the task. It will simply stop the cron trigger from triggering the task
again.
### List Cron Triggers
Retrieves a list of all task cron triggers matching the criteria.
#### Python-Sync
```python
cron_triggers = hatchet.cron.list()
```
#### Python-Async
```python
await hatchet.cron.aio_list()
```
#### Typescript
```typescript
const crons = await hatchet.crons.list({
workflow: simple,
});
```
#### Go
```go
cronList, err := client.Crons().List(context.Background(), rest.CronWorkflowListParams{
AdditionalMetadata: &[]string{"description:Daily cleanup and maintenance tasks"},
})
if err != nil {
return err
}
```
#### Ruby
```ruby
cron_triggers = hatchet.cron.list
```
## Managing Cron Triggers in the Hatchet Dashboard
In the Hatchet Dashboard, you can view and manage cron triggers for your tasks.
Navigate to "Triggers" > "Cron Jobs" in the left sidebar and click "Create Cron Job" at the top right.
You can specify run parameters such as Input, Additional Metadata, and the Expression.
## Cron Considerations
When using cron triggers, there are a few considerations to keep in mind:
1. **Time Zone**: Cron schedules are UTC. Make sure to consider the time zone when defining your cron expressions.
2. **Execution Time**: The actual execution time of a cron-triggered task may vary slightly from the scheduled time. Hatchet makes a best-effort attempt to enqueue the task as close to the scheduled time as possible, but there may be slight delays due to system load or other factors.
3. **Missed Schedules**: If a scheduled task is missed (e.g., due to system downtime), Hatchet will **not** automatically run the missed instances. It will wait for the next scheduled time to trigger the task.
4. **Overlapping Schedules**: If a task is still running when the next scheduled time arrives, Hatchet will start a new instance of the task or respect the [concurrency](/v1/concurrency) policy.
---
# Running Tasks in Bulk
Often you may want to run a task multiple times with different inputs. There is significant overhead (i.e. network roundtrips) to write the task, so if you're running multiple tasks, it's best to use the bulk run methods.
#### Python
You can use the `aio_run_many` method to bulk run a task. This will return a list of results.
```python
greetings = ["Hello, World!", "Hello, Moon!", "Hello, Mars!"]
results = await child_task.aio_run_many(
[
# run each greeting as a task in parallel
child_task.create_bulk_run_item(
input=SimpleInput(message=greeting),
)
for greeting in greetings
]
)
# this will await all results and return a list of results
print(results)
```
> **Info:** `Workflow.create_bulk_run_item` is a typed helper to create the inputs for
> each task.
There are additional bulk methods available on the `Workflow` object.
- `aio_run_many`
- `aio_run_many_no_wait`
And blocking variants:
- `run_many`
- `run_many_no_wait`
As with the run methods, you can call bulk methods from within a task and the runs will be associated with the parent task in the dashboard.
#### Typescript
You can use the `run` method directly to bulk run tasks by passing an array of inputs. This will return a list of results.
```typescript
const res = await simple.run([
{
Message: 'HeLlO WoRlD',
},
{
Message: 'Hello MoOn',
},
]);
// 👀 Access the results of the Task
console.log(res[0].TransformedMessage);
console.log(res[1].TransformedMessage);
```
There are additional bulk methods available on the `Task` object.
- `run`
- `runNoWait`
You can also use `runMany` and `runManyNoWait` for per-run options while keeping the same bulk execution behavior.
```typescript
const runManyRes = await simple.runMany([
{
input: {
Message: 'HeLlO WoRlD',
},
},
{
input: {
Message: 'Hello MoOn',
},
opts: {
priority: 3,
},
},
]);
console.log(runManyRes[0].TransformedMessage);
console.log(runManyRes[1].TransformedMessage);
```
Additional `Task` bulk methods:
- `runMany`
- `runManyNoWait`
As with the run methods, you can call bulk methods on the task fn context parameter within a task and the runs will be associated with the parent task in the dashboard.
```typescript
const parent = hatchet.task({
name: 'simple',
fn: async (input: SimpleInput, ctx) => {
// Bulk run two tasks in parallel
const child = await ctx.bulkRunChildren([
{
workflow: simple,
input: {
Message: 'Hello, World!',
},
},
{
workflow: simple,
input: {
Message: 'Hello, Moon!',
},
},
]);
return {
TransformedMessage: `${child[0].TransformedMessage} ${child[1].TransformedMessage}`,
};
},
});
```
Available bulk methods on the `Context` object are: - `bulkRunChildren` - `bulkRunChildrenNoWait`
#### Go
You can use the `RunMany` method directly on the `Workflow` or `StandaloneTask` instance to bulk run tasks by passing an array of inputs. This will return a list of run IDs.
```go
// Prepare inputs as []RunManyOpt for bulk run
inputs := make([]hatchet.RunManyOpt, len(bulkInputs))
for i, input := range bulkInputs {
inputs[i] = hatchet.RunManyOpt{
Input: input,
}
}
// Run workflows in bulk
ctx := context.Background()
runRefs, err := workflow.RunMany(ctx, inputs)
if err != nil {
log.Fatalf("failed to run bulk workflows: %v", err)
}
```
Additional bulk methods are coming soon for the Go SDK. Join our [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
```ruby
greetings = ["Hello, World!", "Hello, Moon!", "Hello, Mars!"]
results = CHILD_TASK_WF.run_many(
greetings.map do |greeting|
CHILD_TASK_WF.create_bulk_run_item(
input: { "message" => greeting }
)
end
)
puts results
```
---
# Triggering Runs from Events
Run-on-event allows you to trigger one or more tasks when a specific event occurs. This is useful when you need to execute a task in response to an ephemeral event where the result is not important. A few common use cases for event-triggered task runs are:
1. Running a task when an ephemeral event is received, such as a webhook or a message from a queue.
2. When you want to run multiple independent tasks in response to a single event. For instance, if you wanted to run a `send_welcome_email` task, and you also wanted to run a `grant_new_user_credits` task, and a `reward_referral` task, all triggered by the signup. In this case, you might declare all three of those tasks with an event trigger for `user:signup`, and then have them all kick off when that event happens.
> **Info:** Event triggers evaluate tasks to run at the time of the event. If an event is
> received before the task is registered, the task will not be run.
## Declaring Event Triggers
To run a task on an event, you need to declare the event that will trigger the task. This is done by declaring the `on_events` property in the task declaration.
#### Python
```python
EVENT_KEY = "user:create"
SECONDARY_KEY = "foobarbaz"
WILDCARD_KEY = "subscription:*"
class EventWorkflowInput(BaseModel):
should_skip: bool
event_workflow = hatchet.workflow(
name="EventWorkflow",
on_events=[EVENT_KEY, SECONDARY_KEY, WILDCARD_KEY],
input_validator=EventWorkflowInput,
)
```
#### Typescript
```typescript
export const lower = hatchet.workflow({
name: 'lower',
// 👀 Declare the event that will trigger the workflow
onEvents: ['simple-event:create'],
});
```
#### Go
```go
const SimpleEvent = "simple-event:create"
func Lower(client *hatchet.Client) *hatchet.StandaloneTask {
return client.NewStandaloneTask(
"lower", func(ctx hatchet.Context, input EventInput) (*LowerTaskOutput, error) {
return &LowerTaskOutput{
TransformedMessage: strings.ToLower(input.Message),
}, nil
},
hatchet.WithWorkflowEvents(SimpleEvent),
)
}
```
#### Ruby
```ruby
EVENT_KEY = "user:create"
SECONDARY_KEY = "foobarbaz"
WILDCARD_KEY = "subscription:*"
EVENT_WORKFLOW = HATCHET.workflow(
name: "EventWorkflow",
on_events: [EVENT_KEY, SECONDARY_KEY, WILDCARD_KEY]
)
```
> **Info:** Note: Multiple tasks can be triggered by the same event.
> **Info:** As of engine version 0.65.0, Hatchet supports wildcard event triggers using
> the `*` wildcard pattern. For example, you could register `subscription:*` as
> your event key, which would match incoming events like `subscription:create`,
> `subscription:renew`, `subscription:cancel`, and so on.
## Pushing Events to Hatchet
You can push an event to Hatchet by calling the `push` method on the Hatchet event client and providing the event name and payload. Any tasks that have registered an event trigger with a matching event key will be run.
#### Python
```python
hatchet.event.push("user:create", {"should_skip": False})
```
#### Typescript
```typescript
const res = await hatchet.events.push('simple-event:create', {
Message: 'hello',
ShouldSkip: false,
});
```
#### Go
```go
err := client.Events().Push(
context.Background(),
"simple-event:create",
EventInput{
Message: "Hello, World!",
},
)
if err != nil {
return err
}
```
#### Ruby
```ruby
HATCHET.event.push("user:create", { "should_skip" => false })
```
> **Info:** Event triggers evaluate tasks to run at the time of the event. If an event is
> received before the task is registered, the task will not be run.
## Event Filters
Events can be _filtered_ in Hatchet, which allows you to push events to Hatchet and only trigger task runs from them in certain cases. **If you enable filters on a workflow, your workflow will be triggered once for each matching filter on any incoming event with a matching scope** (see [Understanding scopes](#understanding-scopes) below).
### Understanding scopes
Every filter has a required `scope`, which is an arbitrary string that acts as a grouping key for filters. Its purpose is to narrow down the set of candidate filters an event is evaluated against: when you push an event with a `scope`, Hatchet only considers filters whose scope is an exact match for the event's scope, and then triggers the workflow once for each of those filters whose expression evaluates to `true`.
Concretely, for a workflow with event triggers:
- If the workflow has **no filters**, matching events trigger it as usual, and scopes have no effect.
- If the workflow has **one or more filters**, an incoming event will only trigger it if the event's scope exactly matches the scope of at least one filter, _and_ that filter's expression evaluates to `true` for the event. An event pushed without a scope, or with a scope that doesn't match any of the workflow's filters, will not trigger the workflow at all.
Scopes are most useful when you want per-entity filtering, such as in a multi-tenant application where each of your customers should have their own filtering rules. For instance, you might create one filter per customer with a scope like `customer:1234`, each with its own expression and payload. When an event arrives for that customer, you push it with `scope="customer:1234"`, and it's evaluated only against that customer's filters — not against every filter registered for the workflow.
### Basic Usage
There are two ways to create filters in Hatchet.
### Default filters on the workflow
The simplest way to create a filter is to register it declaratively with your workflow when it's created. For example:
#### Python
```python
event_workflow_with_filter = hatchet.workflow(
name="EventWorkflow",
on_events=[EVENT_KEY, SECONDARY_KEY, WILDCARD_KEY],
input_validator=EventWorkflowInput,
default_filters=[
DefaultFilter(
expression="true",
scope="example-scope",
payload={
"main_character": "Anna",
"supporting_character": "Stiva",
"location": "Moscow",
},
)
],
)
```
#### Typescript
```typescript
export const lowerWithFilter = hatchet.workflow({
name: 'lower',
// 👀 Declare the event that will trigger the workflow
onEvents: ['simple-event:create'],
defaultFilters: [
{
expression: 'true',
scope: 'example-scope',
payload: {
mainCharacter: 'Anna',
supportingCharacter: 'Stiva',
location: 'Moscow',
},
},
],
});
```
#### Go
```go
func LowerWithFilter(client *hatchet.Client) *hatchet.StandaloneTask {
return client.NewStandaloneTask(
"lower", accessFilterPayload,
hatchet.WithWorkflowEvents(SimpleEvent),
hatchet.WithDefaultFilters(types.DefaultFilter{
Expression: "true",
Scope: "example-scope",
Payload: map[string]interface{}{
"main_character": "Anna",
"supporting_character": "Stiva",
"location": "Moscow"},
}),
)
}
```
#### Ruby
```ruby
EVENT_WORKFLOW_WITH_FILTER = HATCHET.workflow(
name: "EventWorkflow",
on_events: [EVENT_KEY, SECONDARY_KEY, WILDCARD_KEY],
default_filters: [
Hatchet::DefaultFilter.new(
expression: "true",
scope: "example-scope",
payload: {
"main_character" => "Anna",
"supporting_character" => "Stiva",
"location" => "Moscow"
}
)
]
)
EVENT_WORKFLOW.task(:task) do |input, ctx|
puts "event received"
ctx.filter_payload
end
```
In each of these cases, we register a filter with the workflow. Note that these "declarative" filters are overwritten each time your workflow is updated, so the ids associated with them will not be stable over time. This allows you to modify a filter in-place or remove a filter, and not need to manually delete it over the API.
### Filters feature client
You also can create event filters by using the `filters` clients on the SDKs:
#### Python
```python
hatchet.filters.create(
workflow_id=event_workflow.id,
expression="input.should_skip == false",
# the scope groups filters: only events pushed with a matching
# scope are evaluated against this filter. in a real app, this is
# often an id, e.g. a customer, user, or organization id
scope="foobarbaz",
payload={
"main_character": "Anna",
"supporting_character": "Stiva",
"location": "Moscow",
},
)
```
#### Typescript
```typescript
hatchet.filters.create({
workflowId: lower.id,
expression: 'input.ShouldSkip == false',
scope: 'foobarbaz',
payload: {
main_character: 'Anna',
supporting_character: 'Stiva',
location: 'Moscow',
},
});
```
#### Go
```go
_, err = client.Filters().Create(
context.Background(),
rest.V1CreateFilterRequest{
WorkflowId: uuid.MustParse("bb866b59-5a86-451b-8023-10d451db11d3"),
Expression: "true",
Scope: "example-scope",
},
)
if err != nil {
return err
}
```
#### Ruby
```ruby
HATCHET_CLIENT.filters.create(
workflow_id: EVENT_WORKFLOW.id,
expression: "input.should_skip == false",
scope: "foobarbaz",
payload: {
"main_character" => "Anna",
"supporting_character" => "Stiva",
"location" => "Moscow"
}
)
```
> **Warning:** Note the [`scope`](#understanding-scopes) argument to the filter is required
> **both when creating a filter, and when pushing events**. If the scope on
> filter creation does not match the scope provided when pushing events, the
> filter will not apply.
Then, push an event that uses the filter to determine whether to run. For instance, this run will be skipped, since the payload does not match the expression:
#### Python
```python
hatchet.event.push(
event_key=EVENT_KEY,
payload={
"should_skip": True,
},
scope="foobarbaz",
)
```
#### Typescript
```typescript
hatchet.events.push(
SIMPLE_EVENT,
{
Message: 'hello',
ShouldSkip: true,
},
{
scope: 'foobarbaz',
}
);
```
#### Go
```go
skipPayload := map[string]interface{}{
"shouldSkip": true,
}
skipScope := "foobarbaz"
err = client.Events().Push(
context.Background(),
"simple-event:create",
skipPayload,
v0Client.WithFilterScope(&skipScope),
)
if err != nil {
return err
}
```
#### Ruby
```ruby
HATCHET_CLIENT.event.push(
EVENT_KEY,
{ "should_skip" => true },
scope: "foobarbaz"
)
```
But this one will be triggered since the payload _does_ match the expression:
#### Python
```python
hatchet.event.push(
event_key=EVENT_KEY,
payload={
"should_skip": False,
},
scope="foobarbaz",
)
```
#### Typescript
```typescript
hatchet.events.push(
SIMPLE_EVENT,
{
Message: 'hello',
ShouldSkip: false,
},
{
scope: 'foobarbaz',
}
);
```
#### Go
```go
triggerPayload := map[string]interface{}{
"shouldSkip": false,
}
triggerScope := "foobarbaz"
err = client.Events().Push(
context.Background(),
"simple-event:create",
triggerPayload,
v0Client.WithFilterScope(&triggerScope),
)
if err != nil {
return err
}
```
#### Ruby
```ruby
HATCHET_CLIENT.event.push(
EVENT_KEY,
{ "should_skip" => false },
scope: "foobarbaz"
)
```
> **Info:** In Hatchet, filters are "positive", meaning that we look for _matches_ to the
> filter to determine which tasks to trigger.
### Accessing the filter payload
You can access the filter payload by using the `Context` in the task that was triggered by your event:
#### Python
```python
@event_workflow_with_filter.task()
def filtered_task(input: EventWorkflowInput, ctx: Context) -> None:
print(ctx.filter_payload)
```
#### Typescript
```typescript
lowerWithFilter.task({
name: 'lowerWithFilter',
fn: (input, ctx) => {
ctx.logger.info('filterPayload', { filterPayload: ctx.filterPayload() });
},
});
```
#### Go
```go
func accessFilterPayload(ctx hatchet.Context, input EventInput) (*LowerTaskOutput, error) {
fmt.Println(ctx.FilterPayload())
return &LowerTaskOutput{
TransformedMessage: strings.ToLower(input.Message),
}, nil
}
```
#### Ruby
```ruby
EVENT_WORKFLOW_WITH_FILTER.task(:filtered_task) do |input, ctx|
puts ctx.filter_payload.inspect
end
```
### Advanced Usage
In addition to referencing `input` in the expression (which corresponds to the _event_ payload), you can also reference the following fields:
1. `payload` corresponds to the _filter_ payload (which was part of the request when the filter was created).
2. `additional_metadata` allows for filtering based on `additional_metadata` sent with the event.
3. `event_key` allows for filtering based on the key of the event, such as `user:created`.
---
# Webhooks
Webhooks allow external systems to trigger Hatchet workflows by sending HTTP requests to dedicated endpoints. This enables real-time integration with third-party services like GitHub, Stripe, Slack, or any system that can send webhook events.
## Guides
We have step-by-step guides for the most common webhook integrations:
- [**Stripe**](/cookbooks/webhooks-stripe) — payments, subscriptions, invoices
- [**GitHub**](/cookbooks/webhooks-github) — pull requests, issues, pushes
- [**Slack**](/cookbooks/webhooks-slack) — slash commands, interactive components, event subscriptions
## Creating a webhook
To create a webhook, you'll need to fill out some fields that tell Hatchet how to determine which workflows to trigger from your webhook, and how to validate it when it arrives from the sender. In particular, you'll need to provide the following fields:
#### Name
The **Webhook Name** is tenant-unique (meaning a single tenant can only use each name once), and is used to create the URL for where the incoming webhook request should be sent. For instance, if your tenant id was `d60181b7-da6c-4d4c-92ec-8aa0fc74b3e5` and your webhook name was `my-webhook`, then the URL might look like `https://cloud.onhatchet.run/api/v1/stable/tenants/d60181b7-da6c-4d4c-92ec-8aa0fc74b3e5/webhooks/my-webhook`. Note that you can copy this URL in the dashboard.
#### Source
The **Source** indicates the source of the webhook, which can be a pre-provided one for easy setup like Stripe or Github, or a "generic" one, which lets you configure all of the necessary fields for your webhook integration based on what the webhook sender provides.
#### Event Key Expression
The **Event Key Expression** is a [CEL](https://cel.dev/) expression that you can use to create a dynamic event key from the payload and headers of the incoming webhook. You can either set this to a constant value, like `webhook`, or you could set it to something dynamic using those two options. Some examples:
1. `'stripe:' + input.type` would create event keys where `'stripe:'` is a prefix for all keys indicating the webhook came from Stripe, and `input.type` selects the `type` field off of the webhook payload and uses it to create the final event key. The result might look something like `stripe:payment_intent.created`.
2. `'github:' + headers['x-github-event'] + ':' + input.action` could create a key like `github:star:created`
> **Info:** The result of the event key expression is what Hatchet will use as the event
> key, so you'd need to set a matching event key as a trigger on your workflows
> in order to trigger them from the webhooks you create. For instance, you might
> add `on_events=["stripe:payment_intent.created"]` to listen for payment intent
> created events in the previous example.
#### Scope Expression (Optional)
The **Scope Expression** is an optional [CEL](https://cel.dev/) expression that evaluates to a string used to filter which workflows to trigger. This is useful when you have multiple workflows listening to the same event key but want to route to specific workflows based on the webhook content.
Like the event key expression, you have access to `input` (the webhook payload) and `headers` (the request headers). Some examples:
1. `input.customer_id` would use the customer ID from the payload as the scope
2. `headers['x-organization-id']` would use a header value as the scope
3. `input.metadata.environment` could route to different workflows based on environment
#### Static Payload (Optional)
The **Static Payload** is an optional JSON object that gets merged with the incoming webhook payload before it's passed to your workflows. This is useful for:
- Adding constant metadata to all events from this webhook
- Injecting configuration values that aren't in the original payload
- Overriding specific fields from the incoming payload
> **Info:** When there's a key collision between the incoming webhook payload and the
> static payload, the static payload values take precedence.
For example, if you set a static payload of `{"source": "stripe", "environment": "production"}` and receive a webhook with `{"type": "payment_intent.created", "source": "api"}`, the final payload passed to your workflow would be `{"type": "payment_intent.created", "source": "stripe", "environment": "production"}`.
#### Authentication
Finally, you'll need to specify how Hatchet should authenticate incoming webhook requests. For non-generic sources like Stripe and Github, Hatchet has presets for most of the fields, so in most cases you'd only need to provide a secret.
If you're using a generic source, then you'll need to specify an authentication method (either basic auth, an API key, HMAC-based auth), and provide the required fields (such as a username and password in the basic auth case).
> **Warning:** Hatchet encrypts any secrets you provide for validating incoming webhooks.
The different authentication methods require different fields to be provided:
- **Pre-configured sources** (Stripe, GitHub, Slack): Only require a webhook secret
- **Generic sources** require different fields depending on the selected authentication method:
- **Basic Auth**: Requires a username and password
- **API Key**: Requires header name containing the key on incoming requests, and secret key itself
- **HMAC**: Requires a header name containing the secret on incoming requests, the secret itself, an encoding method (e.g. hex, base64), and an algorithm (e.g. `SHA256`, `SHA1`, etc.).
## Usage
While you're creating your webhook (and also after you've created it), you can copy the webhook URL, which is what you'll provide to the webhook _sender_.
Once you've done that, the last thing to do is register the event keys you want your workers to listen for so that they can be triggered by incoming webhooks.
For examples on how to do this, see the [documentation on event triggers](/v1/events#declaring-event-triggers).
---
## Invoking Tasks From Other Services
While Hatchet recommends importing your workflows and standalone tasks directly to use for triggering runs, this only works in a monorepo or similar setups where you have access to those objects. However, it's common to have a polyrepo, have code written in multiple languages, or otherwise not be able to import your workflows and standalone tasks directly.
Hatchet provides stub tasks for these cases, allowing you to trigger your tasks from anywhere in a type-safe way with only minor code duplication.
### Creating a "Stub" Task on your External Service (Recommended)
The recommended way to trigger a run from a service where you _cannot_ import the workflow or standalone task definition directly is to create a "stub" task or workflow on your external service. This is a Hatchet task or workflow that has the same name and input/output types as the task you want to trigger on your Hatchet worker, but without the function or other configuration.
This allows you to have a polyglot, fully typed interface with full SDK support.
#### Typescript
```typescript
import { hatchet } from '../hatchet-client';
// (optional) Define the input type for the workflow
export type SimpleInput = {
Message: string;
};
// (optional) Define the output type for the workflow
export type SimpleOutput = {
'to-lower': {
TransformedMessage: string;
};
};
// declare the workflow with the same name as the
// workflow name on the worker
export const simple = hatchet.workflow({
name: 'simple',
});
// you can use all the same run methods on the stub
// with full type-safety
simple.run({ Message: 'Hello, World!' });
simple.runNoWait({ Message: 'Hello, World!' });
simple.schedule(new Date(), { Message: 'Hello, World!' });
simple.cron('my-cron', '0 0 * * *', { Message: 'Hello, World!' });
```
#### Python
Consider a task with an implementation like this:
```python
from pydantic import BaseModel
from hatchet_sdk import Context, Hatchet
class TaskInput(BaseModel):
user_id: int
class TaskOutput(BaseModel):
ok: bool
hatchet = Hatchet()
@hatchet.task(name="externally-triggered-task", input_validator=TaskInput)
async def externally_triggered_task(input: TaskInput, ctx: Context) -> TaskOutput:
return TaskOutput(ok=True)
```
To trigger this task from a separate service where the code is not shared, start by defining models that match the input and output types of the task defined above.
```python
class TaskInput(BaseModel):
user_id: int
class TaskOutput(BaseModel):
ok: bool
```
Next, create the stub task.
```python
stub = hatchet.stubs.task(
# make sure the name and schemas exactly match the implementation
name="externally-triggered-task",
input_validator=TaskInput,
output_validator=TaskOutput,
)
```
Finally, use the stub to trigger the underlying task, and (optionally) retrieve the result.
```python
# input type checks properly
result = await stub.aio_run(input=TaskInput(user_id=1234))
# `result.ok` type checks properly
print("Is successful:", result.ok)
```
#### Go
```go
package main
import (
"context"
"fmt"
"log"
hatchet "github.com/hatchet-dev/hatchet/sdks/go"
)
type StubInput struct {
Message string `json:"message"`
}
type StubOutput struct {
Ok bool `json:"ok"`
}
func StubWorkflow(client *hatchet.Client) *hatchet.StandaloneTask {
return client.NewStandaloneTask("stub-workflow", func(ctx hatchet.Context, input StubInput) (StubOutput, error) {
return StubOutput{
Ok: true,
}, nil
})
}
func main() {
client, err := hatchet.NewClient()
if err != nil {
log.Fatalf("failed to create hatchet client: %v", err)
}
task := StubWorkflow(client)
// we are simply running the task here, but it can be implemented in another service / worker
// and in another language with the same name and input-output types
result, err := task.Run(context.Background(), StubInput{Message: "Hello, World!"})
if err != nil {
log.Fatalf("failed to run task: %v", err)
}
fmt.Println(result)
}
```
#### Ruby
Note that this approach requires code duplication, which can break type
safety. For instance, if the input type to your workflow changes, you need to
remember to also change the type passed to the stub. Some ways to mitigate
risks here are helpful comments reminding developers to keep these types in
sync, code generation tools, and end-to-end tests.
---
# Simple Task Retries
Hatchet provides a simple and effective way to handle failures in your tasks using a retry policy. This feature allows you to specify the number of times a task should be retried if it fails, helping to improve the reliability and resilience of your tasks.
> **Info:** Task-level retries can be added to both `Standalone Tasks` and `Workflow
> Tasks`.
## How it works
When a task fails (i.e. throws an error or returns a non-zero exit code), Hatchet can automatically retry the task based on the `retries` configuration defined in the task object. Here's how it works:
1. If a task fails and `retries` is set to a value greater than 0, Hatchet will catch the error and retry the task.
2. The task will be retried up to the specified number of times, with each retry being executed after a short delay to avoid overwhelming the system.
3. If the task succeeds during any of the retries, the task will continue as normal.
4. If the task continues to fail after exhausting all the specified retries, the task will be marked as failed.
This simple retry mechanism can help to mitigate transient failures, such as network issues or temporary unavailability of external services, without requiring complex error handling logic in your task code.
## How to use task-level retries
To enable retries for a task, simply add the `retries` property to the task object in your task definition:
#### Python
```python
@simple_workflow.task(retries=3)
def always_fail(input: EmptyModel, ctx: Context) -> dict[str, str]:
raise Exception("simple task failed")
```
#### Typescript
```typescript
export const retries = hatchet.task({
name: 'retries',
retries: 3,
fn: async (_, ctx) => {
throw new Error('intentional failure');
},
});
```
#### Go
```go
retries := client.NewStandaloneTask("retries-task", func(ctx hatchet.Context, input RetriesInput) (*RetriesResult, error) {
return nil, errors.New("intentional failure")
}, hatchet.WithRetries(3))
```
#### Ruby
```ruby
SIMPLE_RETRY_WORKFLOW.task(:always_fail, retries: 3) do |input, ctx|
raise "simple task failed"
end
```
You can add the `retries` property to any task, and Hatchet will handle the retry logic automatically.
It's important to note that task-level retries are not suitable for all types of failures.
For example, if a task fails due to a programming error or an invalid configuration, retrying the task will likely not resolve the issue.
In these cases, you should fix the underlying problem in your code or configuration rather than relying on retries. See [Bypassing retry logic](#bypassing-retry-logic).
Additionally, if a task interacts with external services or databases, you should ensure that the operation is idempotent (i.e. can be safely repeated without changing the result) before enabling retries. Otherwise, retrying the task could lead to unintended side effects or inconsistencies in your data.
## Accessing the Retry Count in a Running Task
You can access the current retry count on the task's context object:
#### Python
```python
@simple_workflow.task(retries=3)
def fail_twice(input: EmptyModel, ctx: Context) -> dict[str, str]:
if ctx.retry_count < 2:
raise Exception("simple task failed")
return {"status": "success"}
```
#### Typescript
```typescript
export const retriesWithCount = hatchet.task({
name: 'retries-with-count',
retries: 3,
fn: async (_, ctx) => {
// > Get the current retry count
const retryCount = ctx.retryCount();
ctx.logger.info(`Retry count: ${retryCount}`);
if (retryCount < 2) {
throw new Error('intentional failure');
}
return {
message: 'success',
};
},
});
```
#### Go
```go
retriesWithCount := client.NewStandaloneTask("fail-twice-task", func(ctx hatchet.Context, input RetriesWithCountInput) (*RetriesWithCountResult, error) {
// Get the current retry count
retryCount := ctx.RetryCount()
fmt.Printf("Retry count: %d\n", retryCount)
if retryCount < 2 {
return nil, errors.New("intentional failure")
}
return &RetriesWithCountResult{
Message: "success",
}, nil
}, hatchet.WithRetries(3))
```
#### Ruby
```ruby
SIMPLE_RETRY_WORKFLOW.task(:fail_twice, retries: 3) do |input, ctx|
raise "simple task failed" if ctx.retry_count < 2
{ "status" => "success" }
end
```
## Exponential Backoff
Hatchet also supports exponential backoff for retries, which can be useful for handling failures in a more resilient manner. Exponential backoff increases the delay between retries exponentially, giving the failing service more time to recover before the next retry.
#### Python
```python
@backoff_workflow.task(
retries=10,
# 👀 Maximum number of seconds to wait between retries
backoff_max_seconds=10,
# 👀 Factor to increase the wait time between retries.
# This sequence will be 2s, 4s, 8s, 10s, 10s, 10s... due to the maxSeconds limit
backoff_factor=2.0,
)
def backoff_task(input: EmptyModel, ctx: Context) -> dict[str, str]:
if ctx.retry_count < 3:
raise Exception("backoff task failed")
return {"status": "success"}
```
#### Typescript
```typescript
export const withBackoff = hatchet.task({
name: 'with-backoff',
retries: 10,
backoff: {
// 👀 Maximum number of seconds to wait between retries
maxSeconds: 10,
// 👀 Factor to increase the wait time between retries.
// This sequence will be 2s, 4s, 8s, 10s, 10s, 10s... due to the maxSeconds limit
factor: 2,
},
fn: async () => {
throw new Error('intentional failure');
},
});
```
#### Go
```go
withBackoff := client.NewStandaloneTask("with-backoff-task", func(ctx hatchet.Context, input BackoffInput) (*BackoffResult, error) {
return nil, errors.New("intentional failure")
}, hatchet.WithRetries(3), hatchet.WithRetryBackoff(2, 10))
```
#### Ruby
```ruby
BACKOFF_WORKFLOW.task(
:backoff_task,
retries: 10,
# Maximum number of seconds to wait between retries
backoff_max_seconds: 10,
# Factor to increase the wait time between retries.
# This sequence will be 2s, 4s, 8s, 10s, 10s, 10s... due to the maxSeconds limit
backoff_factor: 2.0
) do |input, ctx|
raise "backoff task failed" if ctx.retry_count < 3
{ "status" => "success" }
end
```
## Bypassing Retry logic
The Hatchet SDKs each expose a `NonRetryable` exception, which allows you to bypass pre-configured retry logic for the task. **If your task raises this exception, it will not be retried.** This allows you to circumvent the default retry behavior in instances where you don't want to or cannot safely retry. Some examples in which this might be useful include:
1. A task that calls an external API which returns a 4XX response code.
2. A task that contains a single non-idempotent operation that can fail but cannot safely be rerun on failure, such as a billing operation.
3. A failure that requires manual intervention to resolve.
#### Python
```python
@non_retryable_workflow.task(retries=1)
def should_not_retry(input: EmptyModel, ctx: Context) -> None:
raise NonRetryableException("This task should not retry")
```
#### Typescript
```typescript
const shouldNotRetry = nonRetryableWorkflow.task({
name: 'should-not-retry',
fn: () => {
throw new NonRetryableError('This task should not retry');
},
retries: 1,
});
```
#### Go
```go
retries := client.NewStandaloneTask("non-retryable-task", func(ctx hatchet.Context, input NonRetryableInput) (*NonRetryableResult, error) {
return nil, worker.NewNonRetryableError(errors.New("intentional failure"))
}, hatchet.WithRetries(3))
```
#### Ruby
```ruby
NON_RETRYABLE_WORKFLOW.task(:should_not_retry, retries: 1) do |input, ctx|
raise Hatchet::NonRetryableError, "This task should not retry"
end
NON_RETRYABLE_WORKFLOW.task(:should_retry_wrong_exception_type, retries: 1) do |input, ctx|
raise TypeError, "This task should retry because it's not a NonRetryableError"
end
NON_RETRYABLE_WORKFLOW.task(:should_not_retry_successful_task, retries: 1) do |input, ctx|
# no-op
end
```
In these cases, even though `retries` is set to a non-zero number (meaning the task would ordinarily retry), Hatchet will not retry.
## Python SDK Client Retry Behavior
The retry behavior described above is for task execution inside Hatchet. The Python SDK also has separate retry behavior for certain client-side REST and gRPC calls made by the SDK itself.
These client retries are configured separately from task retries and do not control whether a task is retried after failing in a worker.
> **Info:** Task retries and SDK client retries are separate mechanisms. Task retries
> control whether Hatchet retries a task after task failure. SDK client retries
> control whether the Python SDK retries certain API calls to Hatchet.
### Default client retry behavior
By default, the Python SDK retries certain client calls with exponential backoff, with `max_attempts` defaulting to 5.
**REST API calls**
Error Type, Retried by Default
HTTP 5xx (server errors), Yes
HTTP 404 (not found), Yes
HTTP 429 (too many requests), No
HTTP 400, 401, 403, 409, 422 (client errors), No
Transport errors (timeout, connection, TLS, protocol), No
**gRPC calls**
Status Code, Retried
`UNAVAILABLE`, `DEADLINE_EXCEEDED`, `INTERNAL`, Yes
`RESOURCE_EXHAUSTED`, `ABORTED`, `UNKNOWN`, Yes
`UNIMPLEMENTED`, `NOT_FOUND`, `INVALID_ARGUMENT`, No
`ALREADY_EXISTS`, `UNAUTHENTICATED`, `PERMISSION_DENIED`, No
> **Info:** REST 404 responses are retried by default because some REST reads can observe
> replication lag between the core database and the OLAP database.
### Configuring Python SDK client retries
The Python SDK exposes client retry configuration through `TenacityConfig`, either directly in `ClientConfig` or via environment variables.
```python
import os
from hatchet_sdk import Hatchet
from hatchet_sdk.config import ClientConfig, HTTPMethod, TenacityConfig
hatchet = Hatchet(
config=ClientConfig(
token=os.environ["HATCHET_CLIENT_TOKEN"],
tenacity=TenacityConfig(
max_attempts=5,
retry_429=False,
retry_transport_errors=False,
retry_transport_methods=[HTTPMethod.GET, HTTPMethod.DELETE],
),
)
)
```
Name, Type, Description, Default
`max_attempts`, `int`, Maximum number of retry attempts. Set to 0 to disable retries., `5`
`retry_429`, `bool`, Enable retries for HTTP 429 Too Many Requests responses., `False`
`retry_transport_errors`, `bool`, Enable retries for REST transport-level errors (timeout, connection, TLS)., `False`
`retry_transport_methods`, `list[HTTPMethod]`, HTTP methods to retry on transport errors when `retry_transport_errors` is enabled., `[GET, DELETE]`
You can also configure these via environment variables:
Environment Variable, Description
`HATCHET_CLIENT_TENACITY_MAX_ATTEMPTS`, Maximum retry attempts
`HATCHET_CLIENT_TENACITY_RETRY_429`, Enable 429 retries (`true`/`false`)
`HATCHET_CLIENT_TENACITY_RETRY_TRANSPORT_ERRORS`, Enable transport error retries (`true`/`false`)
### Idempotency considerations
> **Warning:** When `retry_transport_errors` is enabled, only idempotent HTTP methods (`GET`,
> `DELETE`) are retried by default. Non-idempotent methods (`POST`, `PUT`,
> `PATCH`) are excluded because retrying them after a transport error could
> result in duplicate operations if the original request succeeded but the
> response was lost.
You can add non-idempotent methods to `retry_transport_methods`, but only do so if:
1. Your operations are idempotent (for example, because they use idempotency keys), or
2. You understand and accept the risk of duplicate operations
### Retry timing
Python SDK client retries use exponential backoff with jitter. Fine-grained backoff timing is not currently configurable through `TenacityConfig`.
## Go SDK Client Retry Behavior
The retry behavior described above is for task execution inside Hatchet. The Go SDK also retries some REST and gRPC calls that the SDK itself makes to Hatchet.
These SDK client retries are configured separately from task retries. They do not control whether Hatchet retries a task after it fails in a worker.
> **Info:** Task retries and SDK client retries are separate mechanisms. Task retries
> control whether Hatchet retries a task after task failure. SDK client retries
> control whether the Go SDK retries certain API calls to Hatchet.
### Default Go client retry behavior
By default, the Go SDK retries certain client calls with exponential backoff. REST reads use up to 5 total attempts: the initial attempt plus up to 4 retries. gRPC calls keep the existing 5 attempt retry limit.
REST read retries use bounded jittered backoff. When the caller request context has no deadline, each REST attempt uses a response-header timeout without cutting off response body reads. If the caller context already has a deadline, that deadline governs the whole request.
**REST API calls (bodyless `GET` and `HEAD` only)**
Error Type, Retried by Default
HTTP 502, 503, 504 (gateway errors), Yes
HTTP 404 (not found), No
HTTP 429 (too many requests), Yes
HTTP 400, 401, 403, 409, 422 (client errors), No
Transport errors (timeout, connection, TLS, protocol), Yes
For HTTP 429 responses on idempotent reads, the Go SDK honors a valid `Retry-After` header when it fits the client retry cap. When `Retry-After` is missing, invalid, or oversized, it falls back to the same bounded jittered backoff used for other retriable errors.
> **Info:** Unlike the Python SDK, the Go SDK does not retry HTTP 404 responses on REST
> reads in this release. Python retries some 404 reads to account for
> replication lag between the core database and the OLAP database.
**gRPC calls**
Status Code, Retried
`UNAVAILABLE`, `DEADLINE_EXCEEDED`, `INTERNAL`, Yes
`RESOURCE_EXHAUSTED`, Yes
`FAILED_PRECONDITION`, No
`UNIMPLEMENTED`, `NOT_FOUND`, `INVALID_ARGUMENT`, No
`ALREADY_EXISTS`, `UNAUTHENTICATED`, `PERMISSION_DENIED`, No
> **Info:** `FAILED_PRECONDITION` is not retried because Hatchet uses it for non-transient
> control-plane signals such as inactive listeners. The unary interceptor still
> retries all unary RPCs, including writes, in this release.
### Configuring Go SDK client retries
Use environment variables to disable SDK client retries:
Environment Variable, Description
`HATCHET_CLIENT_NO_RETRY`, Disables both REST and gRPC SDK client retries when set to a truthy value.
`HATCHET_CLIENT_NO_GRPC_RETRY`, Legacy gRPC-only retry control. Disables gRPC SDK retries only. REST read retries remain enabled unless `HATCHET_CLIENT_NO_RETRY` is set.
If both variables are set, all SDK client retries are disabled.
### Idempotency considerations
> **Warning:** Go SDK REST retries apply only to bodyless `GET` and `HEAD` requests. `POST`,
> `PUT`, `PATCH`, and `DELETE` requests are never retried by the SDK client in
> this release. Bodied requests are excluded because Go `http.Request` bodies
> are one-shot unless `GetBody` is set or the SDK buffers and rebuilds the body.
> The generated REST clients do not set `GetBody`.
## Conclusion
Hatchet's task-level retry feature is a simple and effective way to handle transient failures in your tasks, improving the reliability and resilience of your tasks. By specifying the number of retries for each task, you can ensure that your tasks can recover from temporary issues without requiring complex error handling logic.
Remember to use retries judiciously and only for tasks that are idempotent. For more advanced retry strategies, such as exponential backoff or circuit breaking, stay tuned for future updates to Hatchet's retry capabilities.
---
# Timeouts in Hatchet
Timeouts are an important concept in Hatchet that allow you to control how long a task is allowed to run before it is considered to have failed. This is useful for ensuring that your tasks don't run indefinitely and consume unnecessary resources. Timeouts in Hatchet are treated as failures and the task will be [retried](/v1/retry-policies) if specified.
There are two types of timeouts in Hatchet:
1. **Scheduling Timeouts** (Default 5m) - the time a task is allowed to wait in the queue before it is cancelled
2. **Execution Timeouts** (Default 60s) - the time a task is allowed to run before it is considered to have failed
## Timeout Format
In Hatchet, timeouts are duration strings: a sequence of decimal numbers, each with an optional fraction and a unit suffix. Valid units are:
- `ms` for milliseconds
- `s` for seconds
- `m` for minutes
- `h` for hours
Components are summed. For example:
- `10s` means 10 seconds
- `4m` means 4 minutes
- `1h` means 1 hour
- `1h30m` means 1 hour 30 minutes
- `42m30s` means 42 minutes 30 seconds
- `1.5h` means 1 hour 30 minutes
A unit is required and values must not be signed; bare numbers like `42` and negative values like `-30s` are rejected.
> **Info:** In the Python SDK, timeouts can also be specified as a `datetime.timedelta`
> object.
### Task-Level Timeouts
You can specify execution and scheduling timeouts for a task using the `execution_timeout` and `schedule_timeout` parameters when creating a task.
#### Ruby
```python
# 👀 Specify an execution timeout on a task
@timeout_wf.task(
execution_timeout=timedelta(seconds=5), schedule_timeout=timedelta(minutes=10)
)
def timeout_task(input: EmptyModel, ctx: Context) -> dict[str, str]:
time.sleep(30)
return {"status": "success"}
```
#### Tab 2
```typescript
export const withTimeouts = hatchet.task({
name: 'with-timeouts',
// time the task can wait in the queue before it is cancelled
scheduleTimeout: '10s',
// time the task can run before it is cancelled
executionTimeout: '10s',
fn: async (input: SimpleInput, ctx) => {
// wait 15 seconds
await sleep(15000);
// get the abort controller
const { abortController } = ctx;
// if the abort controller is aborted, throw an error
if (abortController.signal.aborted) {
throw new Error('cancelled');
}
return {
TransformedMessage: input.Message.toLowerCase(),
};
},
});
```
#### Tab 3
```go
// Task that will timeout - sleeps for 10 seconds but has 3 second timeout
_ = timeoutWorkflow.NewTask("timeout-task",
func(ctx hatchet.Context, input TimeoutInput) (TimeoutOutput, error) {
log.Printf("Starting task that will timeout. Message: %s", input.Message)
// Sleep for 10 seconds (will be interrupted by timeout)
time.Sleep(10 * time.Second)
// This should not be reached due to timeout
log.Println("Task completed successfully (this shouldn't be reached)")
return TimeoutOutput{
Status: "completed",
Completed: true,
}, nil
},
hatchet.WithExecutionTimeout(3*time.Second), // 3 second timeout
)
```
#### Tab 4
```ruby
# Specify an execution timeout on a task
TIMEOUT_WF.task(:timeout_task, execution_timeout: 5, schedule_timeout: 600) do |input, ctx|
sleep 30
{ "status" => "success" }
end
REFRESH_TIMEOUT_WF = HATCHET.workflow(name: "RefreshTimeoutWorkflow")
```
In these tasks, both timeouts are specified, meaning:
1. If the task is not scheduled before the `schedule_timeout` is reached, it will be cancelled.
2. If the task does not complete before the `execution_timeout` is reached (after starting), it will be cancelled.
> **Warning:** A timed out task does not guarantee that the task will be stopped immediately.
> The task will be stopped as soon as the worker is able to stop the task. See
> [cancellation](/v1/cancellation) for more information.
## Refreshing Timeouts
In some cases, you may need to extend the timeout for a task while it is running. This can be done by using the task context.
For example:
#### Ruby
```python
@refresh_timeout_wf.task(execution_timeout=timedelta(seconds=4))
def refresh_task(input: EmptyModel, ctx: Context) -> dict[str, str]:
ctx.refresh_timeout(timedelta(seconds=10))
time.sleep(5)
return {"status": "success"}
```
#### Tab 2
```typescript
export const refreshTimeout = hatchet.task({
name: 'refresh-timeout',
executionTimeout: '10s',
scheduleTimeout: '10s',
fn: async (input: SimpleInput, ctx) => {
// adds 15 seconds to the execution timeout
ctx.refreshTimeout('15s');
await sleep(15000);
// get the abort controller
const { abortController } = ctx;
// now this condition will not be met
// if the abort controller is aborted, throw an error
if (abortController.signal.aborted) {
throw new Error('cancelled');
}
return {
TransformedMessage: input.Message.toLowerCase(),
};
},
});
```
#### Tab 3
```go
// Create workflow with timeout refresh example
refreshTimeoutWorkflow := client.NewWorkflow("refresh-timeout-demo",
hatchet.WithWorkflowDescription("Demonstrates timeout refresh functionality"),
hatchet.WithWorkflowVersion("1.0.0"),
)
// Task that refreshes its timeout to avoid timing out
_ = refreshTimeoutWorkflow.NewTask("refresh-timeout-task",
func(ctx hatchet.Context, input TimeoutInput) (TimeoutOutput, error) {
log.Printf("Starting task with timeout refresh. Message: %s", input.Message)
// Refresh timeout by 10 seconds
log.Println("Refreshing timeout by 10 seconds...")
err := ctx.RefreshTimeout("10s")
if err != nil {
log.Printf("Failed to refresh timeout: %v", err)
return TimeoutOutput{
Status: "failed",
Completed: false,
}, err
}
// Now sleep for 5 seconds (should complete successfully)
log.Println("Sleeping for 5 seconds...")
time.Sleep(5 * time.Second)
log.Println("Task completed successfully after timeout refresh")
return TimeoutOutput{
Status: "completed",
Completed: true,
}, nil
},
hatchet.WithExecutionTimeout(3*time.Second), // Initial 3 second timeout
)
```
#### Tab 4
```ruby
REFRESH_TIMEOUT_WF.task(:refresh_task, execution_timeout: 4) do |input, ctx|
ctx.refresh_timeout(10)
sleep 5
{ "status" => "success" }
end
```
In this example, the task initially would exceed its execution timeout. But before it does, we call the `refreshTimeout` method, which extends the timeout and allows it to complete. Importantly, refreshing a timeout is an additive operation - the new timeout is added to the existing timeout. So for instance, if the task originally had a timeout of `30s` and we call `refreshTimeout("15s")`, the new timeout will be `45s`.
The task timeout can be refreshed multiple times within a task to further extend the timeout as needed.
---
# Cancellation in Hatchet Tasks
Hatchet provides a mechanism for canceling task executions gracefully, allowing you to signal to running tasks that they should stop running. Cancellation can be triggered on graceful termination of a worker or automatically through concurrency control strategies like [`CANCEL_IN_PROGRESS`](/v1/concurrency#cancel-in-progress), which cancels currently running task instances to free up slots for new instances when the concurrency limit is reached.
When a task is canceled, Hatchet sends a cancellation signal to the task. The task can then check for the cancellation signal and take appropriate action, such as cleaning up resources, aborting network requests, or gracefully terminating their execution.
## Cancellation Mechanisms
#### Python
```python
@cancellation_workflow.task()
def check_flag(input: EmptyModel, ctx: Context) -> dict[str, str]:
for i in range(3):
time.sleep(1)
# Note: Checking the status of the exit flag is mostly useful for cancelling
# sync tasks without needing to forcibly kill the thread they're running on.
if ctx.exit_flag:
print("Task has been cancelled")
raise ValueError("Task has been cancelled")
return {"error": "Task should have been cancelled"}
```
```python
@cancellation_workflow.task()
async def self_cancel(input: EmptyModel, ctx: Context) -> dict[str, str]:
await asyncio.sleep(2)
## Cancel the task
await ctx.aio_cancel()
await asyncio.sleep(10)
return {"error": "Task should have been cancelled"}
```
#### Typescript
```typescript
export const cancellation = hatchet.task({
name: 'cancellation',
fn: async (_, ctx) => {
await sleep(10 * 1000);
if (ctx.cancelled) {
throw new Error('Task was cancelled');
}
return {
Completed: true,
};
},
});
```
```typescript
export const abortSignal = hatchet.task({
name: 'abort-signal',
fn: async (_, ctx) => {
try {
const response = await axios.get('https://api.example.com/data', {
signal: ctx.abortController.signal,
});
// Handle the response
} catch (error) {
if (axios.isCancel(error)) {
// Request was canceled
ctx.logger.info('Request canceled');
} else {
// Handle other errors
}
}
},
});
```
#### Go
```go
// Add a long-running task that can be cancelled
_ = workflow.NewTask("long-running-task", func(ctx hatchet.Context, input CancellationInput) (CancellationOutput, error) {
log.Printf("Starting long-running task with message: %s", input.Message)
// Simulate long-running work with cancellation checking
for i := 0; i < 10; i++ {
select {
case <-ctx.Done():
log.Printf("Task cancelled after %d steps", i)
return CancellationOutput{
Status: "cancelled",
Completed: false,
}, nil
default:
log.Printf("Working... step %d/10", i+1)
time.Sleep(1 * time.Second)
}
}
log.Println("Task completed successfully")
return CancellationOutput{
Status: "completed",
Completed: true,
}, nil
}, hatchet.WithExecutionTimeout(30*time.Second))
```
#### Ruby
```ruby
CANCELLATION_WORKFLOW.task(:check_flag) do |input, ctx|
3.times do
sleep 1
# Note: Checking the status of the exit flag is mostly useful for cancelling
# sync tasks without needing to forcibly kill the thread they're running on.
if ctx.cancelled?
puts "Task has been cancelled"
raise "Task has been cancelled"
end
end
{ "error" => "Task should have been cancelled" }
end
```
```ruby
CANCELLATION_WORKFLOW.task(:self_cancel) do |input, ctx|
sleep 2
## Cancel the task
ctx.cancel
sleep 10
{ "error" => "Task should have been cancelled" }
end
```
## Cancellation Best Practices
When working with cancellation in Hatchet tasks, consider the following best practices:
1. **Graceful Termination**: When a task receives a cancellation signal, aim to terminate its execution gracefully. Clean up any resources, abort pending operations, and perform any necessary cleanup tasks before returning from the task function.
2. **Cancellation Checks**: Regularly check for cancellation signals within long-running tasks or loops. This allows the task to respond to cancellation in a timely manner and avoid unnecessary processing.
3. **Cancellation Propagation**: If a task invokes other functions or libraries, consider propagating the cancellation signal to those dependencies. This ensures that cancellation is handled consistently throughout the task.
4. **Error Handling**: Handle cancellation errors appropriately. Distinguish between cancellation errors and other types of errors to provide meaningful error messages and take appropriate actions.
## Additional Features
In addition to the methods of cancellation listed here, Hatchet also supports [bulk cancellation](/v1/bulk-retries-and-cancellations), which allows you to cancel many tasks in bulk using either their IDs or a set of filters, which is often the easiest way to cancel many things at once.
## Conclusion
Cancellation is a powerful feature in Hatchet that allows you to gracefully stop task executions when needed. Remember to follow best practices when implementing cancellation in your tasks, such as graceful termination, regular cancellation checks, handling asynchronous operations, proper error handling, and cancellation propagation.
By incorporating cancellation into your Hatchet tasks and workflows, you can build more resilient and responsive systems that can adapt to changing circumstances and user needs.
---
# Bulk Cancellations and Replays
V1 adds the ability to cancel or replay task runs in bulk, which you can now do either in the Hatchet Dashboard or programmatically via the SDKs and the REST API.
There are two ways of bulk cancelling or replaying tasks in both cases:
1. You can provide a list of task run ids to cancel or replay, which will cancel or replay all of the tasks in the list.
2. You can provide a list of filters, similar to the list of filters on task runs in the Dashboard, and cancel or replay runs matching those filters. For instance, if you wanted to replay all failed runs of a `SimpleTask` from the past fifteen minutes that had the `foo` field in `additional_metadata` set to `bar`, you could apply those filters and replay all of the matching runs.
### Bulk Operations by Run Ids
The first way to bulk cancel or replay runs is by providing a list of run ids. This is the most straightforward way to cancel or replay runs in bulk.
#### Python
> **Info:** In the Python SDK, the mechanics of bulk replaying and bulk cancelling tasks
> are exactly the same. The only change would be replacing e.g.
> `hatchet.runs.bulk_cancel` with `hatchet.runs.bulk_replay`.
First, we'll start by fetching a task via the REST API.
```python
from datetime import datetime, timedelta, timezone
from hatchet_sdk import BulkCancelReplayOpts, Hatchet, RunFilter, V1TaskStatus
hatchet = Hatchet()
workflows = hatchet.workflows.list()
assert workflows.rows
workflow = workflows.rows[0]
```
Now that we have a task, we'll get runs for it, so that we can use them to bulk cancel by run id.
```python
workflow_runs = hatchet.runs.list(workflow_ids=[workflow.metadata.id])
```
And finally, we can cancel the runs in bulk.
```python
workflow_run_ids = [workflow_run.metadata.id for workflow_run in workflow_runs.rows]
bulk_cancel_by_ids = BulkCancelReplayOpts(ids=workflow_run_ids)
hatchet.runs.bulk_cancel(bulk_cancel_by_ids)
```
> **Info:** Note that the Python SDK also exposes async versions of each of these methods:
>
> - `workflows.list` -> `await workflows.aio_list`
> - `runs.list` -> `await runs.aio_list`
> - `runs.bulk_cancel` -> `await runs.aio_bulk_cancel`
#### Typescript
> **Info:** The mechanics of bulk replaying and bulk cancelling tasks are exactly the
> same. The only change would be replacing e.g. `hatchet.runs.cancel` with
> `hatchet.runs.replay`.
First, we'll start by fetching a task via the REST API.
```typescript
const workflows = await hatchet.workflows.list();
if (!workflows.rows?.length) {
throw new Error('no workflows found');
}
const [workflow] = workflows.rows;
```
Now that we have a task, we'll get runs for it, so that we can use them to bulk cancel by run id.
```typescript
const workflowRuns = await hatchet.runs.list({
workflowNames: [workflow.name],
});
```
And finally, we can cancel the runs in bulk.
```typescript
const runIds = workflowRuns.rows?.map((run) => run.metadata.id) ?? [];
// to replay runs by their ids, use `hatchet.runs.replay` instead
await hatchet.runs.cancel({ ids: runIds });
```
#### Go
> **Info:** The mechanics of bulk replaying and bulk cancelling tasks are exactly the
> same. The only change would be replacing e.g. `client.Runs().Cancel` with
> `client.Runs().Replay`.
First, we'll start by fetching a task via the REST API.
```go
client, err := hatchet.NewClient()
if err != nil {
log.Fatalf("failed to create hatchet client: %v", err)
}
ctx := context.Background()
workflows, err := client.Workflows().List(ctx, nil)
if err != nil {
log.Fatalf("failed to list workflows: %v", err)
}
if workflows.Rows == nil || len(*workflows.Rows) == 0 {
log.Fatal("no workflows found")
}
workflow := (*workflows.Rows)[0]
workflowId := uuid.MustParse(workflow.Metadata.Id)
```
Now that we have a task, we'll get runs for it, so that we can use them to bulk cancel by run id.
```go
workflowRuns, err := client.Runs().List(ctx, rest.V1WorkflowRunListParams{
Since: time.Now().Add(-24 * time.Hour),
WorkflowIds: &[]types.UUID{workflowId},
})
if err != nil {
log.Fatalf("failed to list workflow runs: %v", err)
}
```
And finally, we can cancel the runs in bulk.
```go
runIds := make([]types.UUID, len(workflowRuns.Rows))
for i, run := range workflowRuns.Rows {
runIds[i] = uuid.MustParse(run.Metadata.Id)
}
// to replay runs by their ids, use `client.Runs().Replay` with a
// `rest.V1ReplayTaskRequest` instead
_, err = client.Runs().Cancel(ctx, rest.V1CancelTaskRequest{
ExternalIds: &runIds,
})
if err != nil {
log.Fatalf("failed to bulk cancel by run ids: %v", err)
}
```
#### Ruby
> **Info:** The mechanics of bulk replaying and bulk cancelling tasks are exactly the
> same. The only change would be replacing e.g. `hatchet.runs.bulk_cancel` with
> `hatchet.runs.bulk_replay`.
First, we'll start by fetching a task via the REST API.
```ruby
hatchet = Hatchet::Client.new
workflows = hatchet.workflows.list
workflow = workflows.rows.first
```
Now that we have a task, we'll get runs for it, so that we can use them to bulk cancel by run id.
```ruby
workflow_runs = hatchet.runs.list(workflow_ids: [workflow.metadata.id])
```
And finally, we can cancel the runs in bulk.
```ruby
workflow_run_ids = workflow_runs.rows.map { |run| run.metadata.id }
hatchet.runs.bulk_cancel(ids: workflow_run_ids)
```
### Bulk Operations by Filters
The second way to bulk cancel or replay runs is by providing a list of filters. This is the most powerful way to cancel or replay runs in bulk, as it allows you to cancel or replay all runs matching a set of arbitrary filters without needing to provide IDs for the runs in advance.
#### Python
The example below provides some filters you might use to cancel or replay runs in bulk. Importantly, these filters are very similar to the filters you can use in the Hatchet Dashboard to filter which task runs are displaying.
```python
bulk_cancel_by_filters = BulkCancelReplayOpts(
filters=RunFilter(
since=datetime.today() - timedelta(days=1),
until=datetime.now(tz=timezone.utc),
statuses=[V1TaskStatus.RUNNING],
workflow_ids=[workflow.metadata.id],
additional_metadata={"key": "value"},
)
)
hatchet.runs.bulk_cancel(bulk_cancel_by_filters)
```
Running this request will cancel all task runs matching the filters provided.
#### Typescript
The example below provides some filters you might use to cancel or replay runs in bulk. Importantly, these filters are very similar to the filters you can use in the Hatchet Dashboard to filter which task runs are displaying.
```typescript
// to replay runs matching filters, use `hatchet.runs.replay` instead
await hatchet.runs.cancel({
filters: {
since: new Date(Date.now() - 24 * 60 * 60 * 1000),
until: new Date(),
statuses: [V1TaskStatus.RUNNING],
workflowNames: [workflow.name],
additionalMetadata: { key: 'value' },
},
});
```
Running this request will cancel all task runs matching the filters provided.
#### Go
The example below provides some filters you might use to cancel or replay runs in bulk. Importantly, these filters are very similar to the filters you can use in the Hatchet Dashboard to filter which task runs are displaying.
```go
until := time.Now()
// to replay runs matching filters, use `client.Runs().Replay` with a
// `rest.V1ReplayTaskRequest` instead
_, err = client.Runs().Cancel(ctx, rest.V1CancelTaskRequest{
Filter: &rest.V1TaskFilter{
Since: time.Now().Add(-24 * time.Hour),
Until: &until,
Statuses: &[]rest.V1TaskStatus{rest.V1TaskStatusRUNNING},
WorkflowIds: &[]types.UUID{workflowId},
AdditionalMetadata: &[]string{"key:value"},
},
})
if err != nil {
log.Fatalf("failed to bulk cancel by filters: %v", err)
}
```
Running this request will cancel all task runs matching the filters provided.
#### Ruby
The example below provides some filters you might use to cancel or replay runs in bulk. Importantly, these filters are very similar to the filters you can use in the Hatchet Dashboard to filter which task runs are displaying.
```ruby
hatchet.runs.bulk_cancel(
since: Time.now - 86_400,
until_time: Time.now,
statuses: ["RUNNING"],
workflow_ids: [workflow.metadata.id],
additional_metadata: { "key" => "value" }
)
```
Running this request will cancel all task runs matching the filters provided.
# Manual Retries
Hatchet provides a manual retry mechanism that allows you to handle failed task instances flexibly from the Hatchet dashboard.
Navigate to the specific task in the Hatchet dashboard and click on the failed run. From there, you can inspect the details of the run, including the input data and the failure reason for each task.
To retry a failed task, simply click on the task in the run details view and then click the "Replay" button. This will create a new instance of the task, starting from the failed task, and using the same input data as the original run.
Manual retries give you full control over when and how to reprocess failed instances. For example, you may choose to wait until an external service is back online before retrying instances that depend on that service, or you may need to deploy a bug fix to your task code before retrying instances that were affected by the bug.
## A Note on Dead Letter Queues
A dead letter queue (DLQ) is a messaging concept used to handle messages that cannot be processed successfully. In the context of task management, a DLQ can be used to store failed task instances that require manual intervention or further analysis.
While Hatchet does not have a built-in dead letter queue feature, the persistence of failed task instances in the dashboard serves a similar purpose. By keeping a record of failed instances, Hatchet allows you to track and manage failures, perform root cause analysis, and take appropriate actions, such as modifying input data or updating your task code before manually retrying the failed instances.
It's important to note that the term "dead letter queue" is more commonly associated with messaging systems like Apache Kafka or Amazon SQS, where unprocessed messages are automatically moved to a separate queue for manual handling. In Hatchet, the failed instances are not automatically moved to a separate queue but are instead persisted in the dashboard for manual management.
---
# Concurrency Control in Hatchet Tasks
Hatchet provides powerful concurrency control features to help you manage the execution of your tasks. This is particularly useful when you have tasks that may be triggered frequently or have long-running steps, and you want to limit the number of concurrent executions to prevent overloading your system, ensure fairness, or avoid race conditions. Concurrency strategies can be added to both tasks and workflows.
> **Info:** This page will discuss concurrency **keys** often. The key is the result of
> evaluating a [CEL expression](https://celbyexample.com/) that you provide on
> your tasks or workflows. The CEL expression you create can reference the
> `input` to the workflow and the `additional_metadata`.
>
> For instance, the expression `input.user_id + ':' + additional_metadata.foobar` on a workflow run triggered with input `{"user_id": "abc"}` and additional metadata `{"foobar": "bazqux"}` would evaluate to `abc:bazqux`.
## Why use concurrency control?
You should primarily use concurrency control when you need to ensure fair access to resources across your application's users, projects, or organizations. By limiting the number of in-flight tasks for a particular user, you can prevent that user from monopolizing the system.
Concurrency control also lets you limit the number of runs for a workflow globally, if you use a static CEL expression, such as `'global'`. This and [rate limiting](/v1/rate-limits) are the recommended mechanisms for setting per-workflow throughput limits.
> **Info:** Concurrency limits how many runs happen at once. To instead control how much
> of a worker's capacity a single run consumes, so a heavy task takes up more
> slots than a light one, see [Task Slot
> Cost](/v1/advanced-assignment/slot-cost).
## Available Strategies:
- [**Group Round Robin**](#group-round-robin) queues incoming task and workflow runs and only dispatches them to workers and triggers them once an available slot is open.
- [**Cancel In Progress**](#cancel-in-progress) cancels in-progress instances of the task or workflow with matching concurrency keys in order to free up slots for the newly-triggered task or workflow run.
- [**Cancel Newest**](#cancel-newest) cancels any incoming task or workflow runs for a key once the number of runs in a running state for that key has reached a provided limit.
## Group Round Robin
When a new task instance is triggered, the Group Round Robin strategy will:
1. Determine the key that the run belongs to based on the [CEL expression](https://celbyexample.com/) defined in the task or workflow's concurrency configuration.
2. Check if there are any available slots for the computed concurrency key based on the maximum number of concurrent runs allowed by the concurrency configuration.
3. If a slot is available, the new task or workflow starts executing immediately.
4. If no slots are available, the new task or workflow is added to a queue for its key.
5. When a running task instance completes and a slot becomes available for a group, the next queued instance for that group (in round-robin order) is dequeued and starts executing.
Group round robin ensures that task instances are processed fairly across different groups, preventing any one group from monopolizing the available resources. It also helps to reduce latency for instances within each group, as they are processed in a round-robin fashion rather than strictly in the order they were triggered. Group round robin is also useful as a global concurrency control for the maximum number of runs of a single task or workflow that you want executing at any given time, regardless of any grouping. You can set a constant concurrency key as the expression, such as `'*'`, to enable this global concurrency control behavior.
To use this strategy, set the `GROUP_ROUND_ROBIN` limit strategy along with a `max_runs` limit and a key expression:
#### Python
```python
class WorkflowInput(BaseModel):
group: str
concurrency_limit_rr_workflow = hatchet.workflow(
name="ConcurrencyDemoWorkflowRR",
concurrency=ConcurrencyExpression(
expression="input.group",
max_runs=1,
limit_strategy=ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
),
input_validator=WorkflowInput,
)
```
#### Typescript
```typescript
export const simpleConcurrency = hatchet.workflow({
name: 'simple-concurrency',
concurrency: {
maxRuns: 1,
limitStrategy: ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
expression: 'input.GroupKey',
},
});
```
#### Go
```go
var maxRuns int32 = 1
strategy := types.GroupRoundRobin
return client.NewStandaloneTask("simple-concurrency",
func(ctx worker.HatchetContext, input ConcurrencyInput) (*TransformedOutput, error) {
// Random sleep between 200ms and 1000ms
time.Sleep(time.Duration(200+rand.Intn(800)) * time.Millisecond)
return &TransformedOutput{
TransformedMessage: input.Message,
}, nil
},
hatchet.WithWorkflowConcurrency(types.Concurrency{
Expression: "input.GroupKey",
MaxRuns: &maxRuns,
LimitStrategy: &strategy,
}),
)
```
#### Ruby
```ruby
CONCURRENCY_LIMIT_RR_WORKFLOW = HATCHET.workflow(
name: "ConcurrencyDemoWorkflowRR",
concurrency: Hatchet::ConcurrencyExpression.new(
expression: "input.group",
max_runs: 1,
limit_strategy: :group_round_robin
)
)
CONCURRENCY_LIMIT_RR_WORKFLOW.task(:step1) do |input, ctx|
puts "starting step1"
sleep 2
puts "finished step1"
end
```
## Cancel In Progress
When a new task instance is triggered, the Cancel In Progress strategy will:
1. Determine the key that the run belongs to based on the [CEL expression](https://celbyexample.com/) defined in the task or workflow's concurrency configuration.
2. Check if there are any available slots for the computed concurrency key based on the maximum number of concurrent runs allowed by the concurrency configuration.
3. If a slot is available, the new task or workflow starts executing immediately.
4. If no slots are available, a running instance of the task or workflow with the same concurrency key will be cancelled to free up a slot, and the new instance will start executing immediately.
Cancel In Progress ensures that the most recently triggered runs always take priority over older ones, which is useful when newer inputs supersede older ones and in-progress work becomes stale or irrelevant as soon as a newer run arrives. It's particularly well suited for user-facing interactions where only the latest input matters (such as chat messages, form submissions, or search-as-you-type), for resource-intensive tasks where it's more efficient to abandon an old run than wait for it to complete, and for any scenario where you want to prioritize processing the most recent data or events over older ones.
To use this strategy, set the `CANCEL_IN_PROGRESS` limit strategy along with a `max_runs` limit and a key expression:
#### Python
```python
class WorkflowInput(BaseModel):
group: str
concurrency_cancel_in_progress_workflow = hatchet.workflow(
name="ConcurrencyCancelInProgress",
concurrency=ConcurrencyExpression(
expression="input.group",
max_runs=1,
limit_strategy=ConcurrencyLimitStrategy.CANCEL_IN_PROGRESS,
),
input_validator=WorkflowInput,
)
```
#### Typescript
```typescript
export const concurrencyCancelInProgressWorkflow = hatchet.workflow({
name: 'concurrencycancelinprogress',
concurrency: {
expression: 'input.group',
maxRuns: 1,
limitStrategy: ConcurrencyLimitStrategy.CANCEL_IN_PROGRESS,
},
});
```
#### Go
```go
var maxRuns int32 = 1
strategy := types.CancelInProgress
return client.NewStandaloneTask("cancel-in-progress",
func(ctx worker.HatchetContext, input ConcurrencyInput) (*TransformedOutput, error) {
// Random sleep between 200ms and 1000ms
time.Sleep(time.Duration(200+rand.Intn(800)) * time.Millisecond)
return &TransformedOutput{
TransformedMessage: input.Message,
}, nil
},
hatchet.WithWorkflowConcurrency(types.Concurrency{
Expression: "input.GroupKey",
MaxRuns: &maxRuns,
LimitStrategy: &strategy,
}),
)
```
#### Ruby
```ruby
CONCURRENCY_CANCEL_IN_PROGRESS_WORKFLOW = HATCHET.workflow(
name: "ConcurrencyCancelInProgress",
concurrency: Hatchet::ConcurrencyExpression.new(
expression: "input.group",
max_runs: 1,
limit_strategy: :cancel_in_progress
)
)
```
## Cancel Newest
When a new task instance is triggered, the Cancel Newest strategy will:
1. Determine the key that the run belongs to based on the [CEL expression](https://celbyexample.com/) defined in the task or workflow's concurrency configuration.
2. Check if there are any available slots for the computed concurrency key based on the maximum number of concurrent runs allowed by the concurrency configuration.
3. If a slot is available, the new task or workflow starts executing immediately.
4. If no slots are available, the newly triggered run is cancelled immediately, allowing the in-progress runs to continue uninterrupted.
Cancel Newest is the inverse of Cancel In Progress: rather than preempting running work in favor of new arrivals, it protects in-progress runs from being disrupted by allowing them to complete before any new work for the same key is started. This is useful when you want to guarantee that long-running task instances finish without interference, when the cost of restarting work outweighs the value of processing newer inputs, and when you want to prevent a single group's instances from monopolizing the available slots by rejecting excess runs outright instead of queuing them.
To use this strategy, set the `CANCEL_NEWEST` limit strategy along with a `max_runs` limit and a key expression:
#### Python
```python
class WorkflowInput(BaseModel):
group: str
concurrency_cancel_newest_workflow = hatchet.workflow(
name="ConcurrencyCancelNewest",
concurrency=ConcurrencyExpression(
expression="input.group",
max_runs=1,
limit_strategy=ConcurrencyLimitStrategy.CANCEL_NEWEST,
),
input_validator=WorkflowInput,
)
```
#### Typescript
```typescript
export const concurrencyCancelNewestWorkflow = hatchet.workflow({
name: 'concurrencycancelnewest',
concurrency: {
expression: 'input.group',
maxRuns: 1,
limitStrategy: ConcurrencyLimitStrategy.CANCEL_NEWEST,
},
});
```
#### Go
```go
var maxRuns int32 = 1
strategy := types.CancelNewest
return client.NewStandaloneTask("cancel-newest",
func(ctx worker.HatchetContext, input ConcurrencyInput) (*TransformedOutput, error) {
// Random sleep between 200ms and 1000ms
time.Sleep(time.Duration(200+rand.Intn(800)) * time.Millisecond)
return &TransformedOutput{
TransformedMessage: input.Message,
}, nil
},
hatchet.WithWorkflowConcurrency(types.Concurrency{
Expression: "input.GroupKey",
MaxRuns: &maxRuns,
LimitStrategy: &strategy,
}),
)
```
#### Ruby
```ruby
CONCURRENCY_CANCEL_NEWEST_WORKFLOW = HATCHET.workflow(
name: "ConcurrencyCancelNewest",
concurrency: Hatchet::ConcurrencyExpression.new(
expression: "input.group",
max_runs: 1,
limit_strategy: :cancel_newest
)
)
```
## Multiple concurrency strategies
You can also combine multiple concurrency strategies to create a more complex concurrency control system. For example, you can use one group key to represent a specific team, and another group to represent a specific resource in that team, giving you more control over the rate at which tasks are executed.
#### Python
```python
class WorkflowInput(BaseModel):
name: str
digit: str
concurrency_workflow_level_workflow = hatchet.workflow(
name="ConcurrencyWorkflowLevel",
input_validator=WorkflowInput,
concurrency=[
ConcurrencyExpression(
expression="input.digit",
max_runs=DIGIT_MAX_RUNS,
limit_strategy=ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
),
ConcurrencyExpression(
expression="input.name",
max_runs=NAME_MAX_RUNS,
limit_strategy=ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
),
],
)
```
#### Typescript
```typescript
export const multipleConcurrencyKeys = hatchet.workflow({
name: 'simple-concurrency',
concurrency: [
{
maxRuns: 1,
limitStrategy: ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
expression: 'input.Tier',
},
{
maxRuns: 1,
limitStrategy: ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
expression: 'input.Account',
},
],
});
```
#### Go
```go
strategy := types.GroupRoundRobin
var maxRuns int32 = 20
return client.NewStandaloneTask("multi-concurrency",
func(ctx worker.HatchetContext, input ConcurrencyInput) (*TransformedOutput, error) {
// Random sleep between 200ms and 1000ms
time.Sleep(time.Duration(200+rand.Intn(800)) * time.Millisecond)
return &TransformedOutput{
TransformedMessage: input.Message,
}, nil
},
hatchet.WithWorkflowConcurrency(
types.Concurrency{
Expression: "input.Tier",
MaxRuns: &maxRuns,
LimitStrategy: &strategy,
}, types.Concurrency{
Expression: "input.Account",
MaxRuns: &maxRuns,
LimitStrategy: &strategy,
},
),
)
```
#### Ruby
```ruby
CONCURRENCY_WORKFLOW_LEVEL_WORKFLOW = HATCHET.workflow(
name: "ConcurrencyWorkflowLevel",
concurrency: [
Hatchet::ConcurrencyExpression.new(
expression: "input.digit",
max_runs: DIGIT_MAX_RUNS_WL,
limit_strategy: :group_round_robin
),
Hatchet::ConcurrencyExpression.new(
expression: "input.name",
max_runs: NAME_MAX_RUNS_WL,
limit_strategy: :group_round_robin
)
]
)
CONCURRENCY_WORKFLOW_LEVEL_WORKFLOW.task(:task_1) do |input, ctx|
sleep SLEEP_TIME_WL
end
CONCURRENCY_WORKFLOW_LEVEL_WORKFLOW.task(:task_2) do |input, ctx|
sleep SLEEP_TIME_WL
end
```
---
# Rate Limiting Step Runs in Hatchet
Hatchet allows you to enforce rate limits on task runs, enabling you to control the rate at which your service runs consume resources, such as external API calls, database queries, or other services. By defining rate limits, you can prevent task runs from exceeding a certain number of requests per time window (e.g., per second, minute, or hour), ensuring efficient resource utilization and avoiding overloading external services.
The state of active rate limits can be viewed in the dashboard in the `Rate Limit` resource tab.
## Dynamic vs Static Rate Limits
Hatchet offers two patterns for Rate Limiting task runs:
1. [Dynamic Rate Limits](#dynamic-rate-limits): Allows for complex rate limiting scenarios, such as per-user limits, by using `input` or `additional_metadata` keys to upsert a limit at runtime.
2. [Static Rate Limits](#static-rate-limits): Allows for simple rate limiting for resources known prior to runtime (e.g., external APIs).
## Dynamic Rate Limits
Dynamic rate limits are ideal for complex scenarios where rate limits need to be partitioned by resources that are only known at runtime.
This pattern is especially useful for:
1. Rate limiting individual users or tenants
2. Implementing variable rate limits based on subscription tiers or user roles
3. Dynamically adjusting limits based on real-time system load or other factors
### How It Works
1. Define the dynamic rate limit key with a CEL (Common Expression Language) Expression on the key, referencing either `input` or `additional_metadata`.
2. Provide this key as part of the workflow trigger or event `input` or `additional_metadata` at runtime.
3. Hatchet will create or update the rate limit based on the provided key and enforce it for the step run.
> **Info:** Note: Dynamic keys are a shared resource, this means the same rendered cel on
> multiple steps will be treated as one global rate limit.
### Declaring and Consuming Dynamic Rate Limits
#### Ruby
> Note: `dynamic_key` must be a CEL expression. `units` and `limits` can be either an integer or a CEL expression.
We can add one or more rate limits to a task by adding the `rate_limits` configuration to the task definition.
```python
@rate_limit_workflow.task(
rate_limits=[
RateLimit(
dynamic_key="input.user_id",
units=1,
limit=10,
duration=RateLimitDuration.MINUTE,
)
]
)
def step_2(input: RateLimitInput, ctx: Context) -> None:
print("executed step_2")
```
#### Tab 2
> Note: `dynamicKey` must be a CEL expression. `units` and `limit` can be either an integer or a CEL expression.
We can add one or more rate limits to a task by adding the `rate_limits` configuration to the task definition.
```typescript
const task2 = hatchet.task({
name: 'task2',
fn: (input: { userId: string }, ctx) => {
ctx.logger.info(`executed task2 for user: ${input.userId}`);
},
rateLimits: [
{
dynamicKey: 'input.userId',
units: 1,
limit: 10,
duration: RateLimitDuration.MINUTE,
},
],
});
```
#### Tab 3
> Note: Go requires both a key and KeyExpr be set and the LimitValueExpr must be a CEL.
```go
userUnits := 1
userLimit := "10"
duration := types.Minute
dynamicTask := client.NewStandaloneTask("task2",
func(ctx hatchet.Context, input APIRequest) (string, error) {
log.Printf("executed task2 for user: %s", input.UserID)
return "completed", nil
},
hatchet.WithRateLimits(&types.RateLimit{
Key: "input.userId",
Units: &userUnits,
LimitValueExpr: &userLimit,
Duration: &duration,
}),
)
```
#### Tab 4
```ruby
RATE_LIMIT_WORKFLOW.task(
:step_2,
rate_limits: [
Hatchet::RateLimit.new(
dynamic_key: "input.user_id",
units: 1,
limit: 10,
duration: :minute
)
]
) do |input, ctx|
puts "executed step_2"
end
```
## Static Rate Limits
Static Rate Limits (formerly known as Global Rate Limits) are defined as part of your worker startup lifecycle prior to runtime. This model provides a single "source of truth" for pre-defined resources such as:
1. External API resources that have a rate limit across all users or tenants
2. Database connection pools with a maximum number of concurrent connections
3. Shared computing resources with limited capacity
### How It Works
1. Declare static rate limits using the `put_rate_limit` method in the `Admin` client before starting your worker.
2. Specify the units of consumption for a specific rate limit key in each step definition using the `rate_limits` configuration.
3. Hatchet enforces the defined rate limits by tracking the number of units consumed by each step run across all workflow runs.
If a step run exceeds the rate limit, Hatchet re-queues the step run until the rate limit is no longer exceeded.
### Declaring Static Limits
Define the static rate limits that can be consumed by any step run across all workflow runs using the `put_rate_limit` method in the `Admin` client within your code.
#### Ruby
```python
RATE_LIMIT_KEY = "test-limit"
hatchet.rate_limits.put(RATE_LIMIT_KEY, 2, RateLimitDuration.SECOND)
```
#### Tab 2
{" "}
```typescript
hatchet.ratelimits.upsert({
key: 'api-service-rate-limit',
limit: 10,
duration: RateLimitDuration.SECOND,
});
```
#### Tab 3
```go
err = client.RateLimits().Upsert(features.CreateRatelimitOpts{
Key: RATE_LIMIT_KEY,
Limit: 10,
Duration: types.Second,
})
if err != nil {
log.Fatalf("failed to create rate limit: %v", err)
}
```
#### Tab 4
```ruby
def main
HATCHET.rate_limits.put(RATE_LIMIT_KEY, 2, :second)
worker = HATCHET.worker(
"rate-limit-worker", slots: 10, workflows: [RATE_LIMIT_WORKFLOW]
)
worker.start
end
```
### Consuming Static Rate Limits
With your rate limit key defined, specify the units of consumption for a specific key in each step definition by adding the `rate_limits` configuration to your step definition in your workflow.
#### Ruby
```python
RATE_LIMIT_KEY = "test-limit"
@rate_limit_workflow.task(rate_limits=[RateLimit(static_key=RATE_LIMIT_KEY, units=1)])
def step_1(input: RateLimitInput, ctx: Context) -> None:
print("executed step_1")
```
#### Tab 2
```typescript
const RATE_LIMIT_KEY = 'api-service-rate-limit';
const task1 = hatchet.task({
name: 'task1',
rateLimits: [
{
staticKey: RATE_LIMIT_KEY,
units: 1,
},
],
fn: (_input, ctx) => {
ctx.logger.info('executed task1');
},
});
```
#### Tab 3
```go
units := 1
staticTask := client.NewStandaloneTask("task1",
func(ctx hatchet.Context, input APIRequest) (string, error) {
log.Println("executed task1")
return "completed", nil
},
hatchet.WithRateLimits(&types.RateLimit{
Key: RATE_LIMIT_KEY,
Units: &units,
}),
)
```
#### Tab 4
```ruby
RATE_LIMIT_KEY = "test-limit"
RATE_LIMIT_WORKFLOW.task(
:step_1,
rate_limits: [Hatchet::RateLimit.new(static_key: RATE_LIMIT_KEY, units: 1)]
) do |input, ctx|
puts "executed step_1"
end
```
### Limiting Workflow Runs
To rate limit an entire workflow run, it's recommended to specify the rate limit configuration on the entry step (i.e., the first step in the workflow). This will gate the execution of all downstream steps in the workflow.
---
# Assigning priority to tasks in Hatchet
Hatchet allows you to assign different `priority` values to your tasks depending on how soon you want them to run. `priority` can be set to either `1`, `2`, or `3`, (`low`, `medium`, and `high`, respectively) with relatively higher values resulting in that task being picked up before others of the same type. **By default, runs in Hatchet have a priority of 1 (low) unless otherwise specified.**
Priority only affects multiple runs of a _single_ workflow. If you have two different workflows (A and B) and set A to globally have a priority of 3, and B to globally have a priority of 1, this does _not_ guarantee that if there is one task from A and one from B in the queue, that A's task will be run first.
However, _within_ A, if you enqueue one task with priority 3 and one with priority 1, the priority 3 task will be run first.
A couple of common use cases for assigning priorities are things like:
1. Having high-priority (e.g. paying, new, etc.) customers be prioritized over lower-priority ones, allowing them to get faster turnaround times on their tasks.
2. Having tasks triggered via your API run with higher priority than the same tasks triggered by a cron.
## Setting priority for a task or workflow
There are a few different ways to set priorities for tasks or workflows in Hatchet.
### Workflow-level default priority
First, you can set a default priority at the workflow level:
#### Ruby
```python
DEFAULT_PRIORITY = Priority.LOW
SLEEP_TIME = 0.25
priority_workflow = hatchet.workflow(
name="PriorityWorkflow",
default_priority=DEFAULT_PRIORITY,
)
```
#### Tab 2
```typescript
export const priorityWf = hatchet.workflow({
name: 'priority-wf',
defaultPriority: Priority.LOW,
});
```
#### Tab 3
```go
workflow := client.NewWorkflow(
"priority",
hatchet.WithWorkflowDefaultPriority(features.RunPriorityLow),
)
```
#### Tab 4
```ruby
DEFAULT_PRIORITY = 1
SLEEP_TIME = 0.25
PRIORITY_WORKFLOW = HATCHET.workflow(
name: "PriorityWorkflow",
default_priority: DEFAULT_PRIORITY
)
PRIORITY_WORKFLOW.task(:priority_task) do |input, ctx|
puts "Priority: #{ctx.priority}"
sleep SLEEP_TIME
end
```
This will assign the same default priority to all runs of this workflow (and all of the workflow's corresponding tasks), but will have no effect without also setting run-level priorities, since every run will use the same default.
### Priority-on-trigger
When you trigger a run, you can set the priority of the triggered run to override its default priority.
#### Ruby
```python
low_prio = priority_workflow.run(
## 👀 Adding priority and key to metadata to show them in the dashboard
priority=Priority.LOW,
additional_metadata={"priority": "low", "key": 1},
wait_for_result=False,
)
high_prio = priority_workflow.run(
## 👀 Adding priority and key to metadata to show them in the dashboard
priority=Priority.HIGH,
additional_metadata={"priority": "high", "key": 1},
wait_for_result=False,
)
```
#### Tab 2
```typescript
const run = priority.run(new Date(Date.now() + 60 * 60 * 1000), { priority: Priority.HIGH });
```
#### Tab 3
```go
ref, err := client.RunNoWait(
context.Background(),
workflow.GetName(),
PriorityInput{},
hatchet.WithRunPriority(features.RunPriorityLow),
)
if err != nil {
return err
}
```
#### Tab 4
```ruby
low_prio = PRIORITY_WORKFLOW.run_no_wait(
{},
options: Hatchet::TriggerWorkflowOptions.new(
priority: 1,
additional_metadata: { "priority" => "low", "key" => 1 }
)
)
high_prio = PRIORITY_WORKFLOW.run_no_wait(
{},
options: Hatchet::TriggerWorkflowOptions.new(
priority: 3,
additional_metadata: { "priority" => "high", "key" => 1 }
)
)
```
Similarly, you can also assign a priority to scheduled and cron workflows.
#### Ruby
```python
schedule = priority_workflow.schedule(
run_at=datetime.now(tz=timezone.utc) + timedelta(minutes=1),
priority=Priority.HIGH,
)
cron = priority_workflow.create_cron(
cron_name="my-scheduled-cron",
expression="0 * * * *",
priority=Priority.HIGH,
)
```
#### Tab 2
```typescript
const scheduled = priority.schedule(
new Date(Date.now() + 60 * 60 * 1000),
{},
{ priority: Priority.HIGH }
);
const delayed = priority.delay(60 * 60 * 1000, {}, { priority: Priority.HIGH });
const cron = priority.cron(
`daily-cron-${Math.random()}`,
'0 0 * * *',
{},
{ priority: Priority.HIGH }
);
```
#### Tab 3
```go
priority := features.RunPriorityHigh
schedule, err := client.Schedules().Create(
context.Background(),
workflow.GetName(),
features.CreateScheduledRunTrigger{
Priority: &priority,
},
)
if err != nil {
return err
}
cron, err := client.Crons().Create(
context.Background(),
workflow.GetName(),
features.CreateCronTrigger{
Priority: &priority,
},
)
if err != nil {
return err
}
```
#### Tab 4
```ruby
schedule = PRIORITY_WORKFLOW.schedule(
Time.now + 60,
options: Hatchet::TriggerWorkflowOptions.new(priority: 3)
)
cron = PRIORITY_WORKFLOW.create_cron(
"my-scheduled-cron",
"0 * * * *",
input: {},
)
```
In these cases, the priority set on the trigger will override the default priority, so these runs will be processed ahead of lower-priority ones.
---
# Idempotency
> **Info:** Idempotency is currently in beta and may be subject to change.
If you need to prevent more than one run of a task from occurring within a given time window, for instance because of duplicate event sends from a webhook that can trigger duplicate runs in Hatchet, you can achieve this by adding **idempotency** configuration to your workflow or standalone task.
## Types of Idempotency in Hatchet
Hatchet supports two different types of idempotency behavior, which you can choose between depending on the needs of your application:
1. **TTL-based** idempotency, which says that there can only be one run for a given idempotency key within a specified amount of time after the first run for that key is seen.
2. **Status-based** idempotency, which clears the idempotency key when the run that claimed it reaches a terminal status (success, failure, or cancellation). This acts similarly to `CANCEL_NEWEST` concurrency, where any time there's a running task holding an idempotency key, any incoming tasks with the same key will be dropped. But once the running task reaches a terminal state, a new one that's triggered can immediately claim the key once again, with no fixed wait time.
## The Idempotency Key Expression
Every idempotency configuration, regardless of strategy, requires an **expression**, which is a CEL expression that's evaluated against the input and additional metadata of the run that's about to be triggered to produce the idempotency key. Two runs with the same computed key are considered duplicates.
> **Warning:** The idempotency key expression must evaluate to a string.
## TTL-based Idempotency
TTL-based idempotency requires two parameters: the **expression** described above, and a **TTL**, which determines how long the key should live for. Only one run for a given key will occur in the time window from when the first trigger comes in until the TTL expires.
#### Python
```python
EVENT_KEY = "idempotency:example"
class IdempotencyInput(BaseModel):
id: str
desired_status: Literal["success", "cancel", "fail"] = "success"
@hatchet.task(
idempotency=TTLBasedIdempotencyConfig(
key_expression="input.id", ttl=timedelta(minutes=1)
),
input_validator=IdempotencyInput,
on_events=[EVENT_KEY],
)
async def idempotent_task(input: IdempotencyInput, ctx: Context) -> dict[str, str]:
return {"result": f"Hello, world from task {input.id}"}
```
#### Typescript
```typescript
export const idempotentTask = hatchet.task({
name: 'ts-e2e-idempotent-task',
idempotency: {
strategy: 'ttl',
expression: 'input.id',
ttlMs: 60_000,
},
onEvents: [EVENT_KEY],
fn: async (input) => {
return { result: `Hello, world from task ${input.id}` };
},
});
```
#### Go
```go
func IdempotentTask(client *hatchet.Client) *hatchet.StandaloneTask {
return client.NewStandaloneTask(
"idempotent-task",
func(ctx hatchet.Context, input IdempotencyInput) (*IdempotencyOutput, error) {
return &IdempotencyOutput{
Result: fmt.Sprintf("Hello, world from task %s", input.ID),
}, nil
},
hatchet.WithWorkflowIdempotency(hatchet.IdempotencyConfig{
Expression: "input.id",
TTL: time.Minute,
}),
)
}
```
#### Ruby
```ruby
IDEMPOTENT_TASK = HATCHET.task(
name: 'ruby-e2e-idempotent-task',
idempotency: Hatchet::TTLBasedIdempotencyConfig.new(expression: 'input.id', ttl_ms: 60_000),
on_events: [EVENT_KEY]
) do |input, _ctx|
{ 'result' => "Hello from task #{input['id']}" }
end
IDEMPOTENT_TASK_SHORT_WINDOW = HATCHET.task(
name: 'ruby-e2e-idempotent-task-short-window',
idempotency: Hatchet::TTLBasedIdempotencyConfig.new(expression: 'input.id', ttl_ms: 2_000)
) do |input, _ctx|
{ 'result' => "Hello from task #{input['id']}" }
end
```
The TTL window is _sliding_: each accepted run resets the clock. For instance, if you trigger a workflow at `00:00:00 UTC` (midnight) with a TTL of five minutes, and then the same workflow is triggered again with the same inputs and metadata at `00:02:00 UTC`, `00:04:00 UTC`, and `00:06:00 UTC`, only the first one (at midnight) and the final one (at `00:06:00 UTC`) will run. After the second run occurs, the lock on the key will be held until `00:11:00 UTC` (five minutes after the final run was triggered).
TTL-based idempotency is a good fit when you want to _debounce_ duplicate triggers over a fixed period of time, regardless of how long the run itself takes.
## Status-based Idempotency
Status-based idempotency keeps the idempotency key claimed only while the run that owns it is still active. Once that run reaches a terminal status (success, failure, or cancellation), the key is released immediately, and the next trigger with the same key can claim it and start a fresh run with no fixed wait time.
This behaves similarly to `CANCEL_NEWEST` concurrency: while a run is holding the key, any incoming run with the same key is dropped (raising an idempotency collision), but once the running task finishes, a new one can immediately take its place.
Instead of a TTL, status-based idempotency takes a **fallback TTL**. Because the key is normally released when the run reaches a terminal state, the fallback TTL exists only as a safety net: it caps the longest the key can remain claimed if, for some reason, the run never reaches a terminal status. You should generally set the fallback TTL comfortably longer than you expect the run to take.
#### Python
```python
@hatchet.task(
idempotency=StatusBasedIdempotencyConfig(
key_expression="input.id", fallback_ttl=timedelta(seconds=10)
),
input_validator=IdempotencyInput,
)
async def idempotent_status_based_task(
input: IdempotencyInput,
ctx: Context,
) -> dict[str, str]:
if input.desired_status == "success":
await asyncio.sleep(2)
return {"result": f"Hello, world from task {input.id}"}
if input.desired_status == "fail":
await asyncio.sleep(2)
raise Exception(f"Task {input.id} failed as requested.")
if input.desired_status == "cancel":
await asyncio.sleep(1)
await ctx.aio_cancel()
for _ in range(10):
await asyncio.sleep(1)
raise Exception(f"Task {input.id} should have been cancelled, but was not.")
```
#### Typescript
```typescript
export const idempotentStatusBasedTask = hatchet.task({
name: 'ts-e2e-idempotent-status-based-task',
idempotency: {
strategy: 'status',
expression: 'input.id',
fallbackTtlMs: 10_000,
},
fn: async (input) => {
return { result: `Hello, world from task ${input.id}` };
},
});
```
#### Go
```go
func IdempotentStatusBasedTask(client *hatchet.Client) *hatchet.StandaloneTask {
return client.NewStandaloneTask(
"idempotent-status-based-task",
func(ctx hatchet.Context, input IdempotencyInput) (*IdempotencyOutput, error) {
return &IdempotencyOutput{
Result: fmt.Sprintf("Hello, world from task %s", input.ID),
}, nil
},
hatchet.WithWorkflowIdempotency(hatchet.IdempotencyConfig{
Expression: "input.id",
Method: hatchet.IdempotencyMethodStatus,
TTL: 10 * time.Second,
}),
)
}
```
#### Ruby
```ruby
IDEMPOTENT_STATUS_BASED_TASK = HATCHET.task(
name: 'ruby-e2e-idempotent-status-based-task',
idempotency: Hatchet::StatusBasedIdempotencyConfig.new(expression: 'input.id', fallback_ttl_ms: 10_000)
) do |input, _ctx|
{ 'result' => "Hello from task #{input['id']}" }
end
```
For example, if you trigger a run at `00:00:00 UTC` that takes thirty seconds to complete, any duplicate triggers that come in before `00:00:30 UTC` will collide and be rejected. But a duplicate that arrives at `00:00:31 UTC`, just after the first run has finished, will be accepted and start a new run, because the key was released as soon as the first run reached its terminal status.
Status-based idempotency is a good fit when you want to guarantee that only one run for a given key is _in flight_ at any moment, without imposing a cooldown after it completes.
## Handling Collisions
When a collision occurs, the engine will reject the workflow run, and, if the run was triggered from an SDK, then the SDK will raise an exception indicating that there was an idempotency collision. This exception will contain the id of the workflow run that already existed that had claimed the key already, so you can retrieve its output if you like.
#### Python
```python
ref_1 = await idempotent_task.aio_run(
input=IdempotencyInput(id="123"),
wait_for_result=False,
)
try:
ref_2 = await idempotent_task.aio_run(
input=IdempotencyInput(id="123"),
wait_for_result=False,
)
run_id_2 = ref_2.workflow_run_id
except IdempotencyCollisionError as e:
print(
f"Run with external ID {e.existing_run_external_id} already exists for this idempotency key"
)
run_id_2 = e.existing_run_external_id
res_1 = await ref_1.aio_result()
res_2 = await idempotent_task.aio_get_result(run_id_2)
assert res_1 == res_2
assert ref_1.workflow_run_id == run_id_2
```
#### Typescript
```typescript
const ref1 = await idempotentTask.runNoWait({ id: '123' });
let runId2: string;
try {
const ref2 = await idempotentTask.runNoWait({ id: '123' });
runId2 = await ref2.getWorkflowRunId();
} catch (e) {
if (e instanceof IdempotencyCollisionError) {
console.log(
`Run with external ID ${e.existingRunExternalId} already exists for this idempotency key`
);
runId2 = e.existingRunExternalId;
} else {
throw e;
}
}
const res1 = await ref1.result();
console.log(`Result: ${JSON.stringify(res1)}, run ID: ${runId2}`);
```
#### Go
```go
ref1, err := idempotentTask.RunNoWait(ctx, IdempotencyInput{ID: "123"})
if err != nil {
log.Fatalf("unexpected error on first run: %v", err)
}
ref2, err := idempotentTask.RunNoWait(ctx, IdempotencyInput{ID: "123"})
var runID2 string
if err != nil {
if idempErr, ok := hatchet.IsIdempotencyCollisionError(err); ok {
fmt.Printf("Run %s already exists for this idempotency key\n", idempErr.ExistingRunExternalId)
runID2 = idempErr.ExistingRunExternalId
} else {
log.Fatalf("unexpected error on second run: %v", err)
}
} else {
runID2 = ref2.RunId
}
```
#### Ruby
```ruby
first_ref = IDEMPOTENT_TASK.run_no_wait({ 'id' => '123' })
second_run_id = begin
second_ref = IDEMPOTENT_TASK.run_no_wait({ 'id' => '123' })
second_ref.workflow_run_id
rescue Hatchet::IdempotencyCollisionError => e
puts "Run #{e.existing_run_external_id} already exists for this idempotency key"
e.existing_run_external_id
end
```
In other cases, such as triggering by events, the idempotency collision will be swallowed, and no additional runs will be created, but the event will still be ingested correctly without an error being raised.
---
# Batch Tasks
> **Warning:** Batch tasks are in beta and may change in future releases.
For tasks that do not need to be executed immediately, you can use batch tasks
to automatically accumulate tasks and execute them in a single batch. The input
to a batch task is a map that associates the ID of the originating task with
it's input. The output *must* be the same shape, using the same keys, to send
outputs back to the original call site (unless output broadcasting is used, see
[Broadcasting outputs](#broadcasting-outputs)). For the caller of a batch task,
the batched execution is hidden. It is called, and receives output, the same way
as a normal task. Though each batch will be executed in one handler function,
the call site only receives output corresponding to its input, not the output
for the entire batch.
## Batch flushing behavior
Batches will accumulate tasks continuously, preventing them from executing until one of the following conditions is true:
- The max size for the batch is reached.
- The maximum interval has elapsed since the last batch flush.
- The size of the payloads for the buffered tasks has exceeded the 4Mb gRPC message limit.
In the following example, the batch task will be executed once 3 tasks have accumulated, or every 200ms, whichever comes first.
#### Python
```python
@hatchet.batch_task(
batch_max_size=3,
batch_max_interval=timedelta(milliseconds=200),
input_validator=SimpleInput,
)
async def batch_simple(
tasks: dict[BatchMemberId, SimpleInput], context: Context
) -> dict[BatchMemberId, SimpleOutput]:
return {
id: SimpleOutput(transformed_message=inp.message.upper())
for id, inp in tasks.items()
}
```
#### Typescript
```typescript
export const batchSimple = hatchet.batchTask({
name: 'batch-simple',
batch: { maxSize: 3, maxInterval: 200 },
fn: async (tasks: Record) => {
const out: Record = {};
Object.entries(tasks).forEach(([id, input]) => {
out[id] = { transformed_message: input.message.toUpperCase() };
});
return out;
},
});
```
#### Go
```go
batchSimple := client.NewStandaloneBatchTask("batch-simple",
func(ctx hatchet.Context, tasks map[string]SimpleInput) (map[string]SimpleOutput, error) {
out := make(map[string]SimpleOutput, len(tasks))
for id, inp := range tasks {
out[id] = SimpleOutput{TransformedMessage: strings.ToUpper(inp.Message)}
}
return out, nil
},
hatchet.BatchConfig{
MaxSize: 3,
MaxInterval: durationPtr(200 * time.Millisecond),
},
)
```
#### Ruby
```ruby
BATCH_SIMPLE = HATCHET.batch_task(
name: 'ruby-e2e-batch-simple',
batch: Hatchet::BatchTaskConfig.new(max_size: 3, max_interval_ms: 200)
) do |inputs, _ctx|
inputs.transform_values { |input| { 'transformed_message' => input['message'].upcase } }
end
```
## Batch keys
To allow for accumulation of multiple batches simultaneously using the same workflow, batch keys can be used.
For example, if you had multiple tenants, and wanted to ensure that each batch execution did not mix inputs from multiple
tenants, batch keys can be used to partition the inputs. In the following example, the batches will flush independently based on the `group` value.
#### Python
```python
@hatchet.batch_task(
batch_max_size=2,
batch_max_interval=timedelta(milliseconds=200),
batch_group_key="input.group",
input_validator=KeyedInput,
)
async def batch_keyed(
tasks: dict[BatchMemberId, KeyedInput], context: Context
) -> dict[BatchMemberId, KeyedOutput]:
unique_keys = len({inp.group for _, inp in tasks.items()})
return {
id: KeyedOutput(
batch_key=inp.group,
batch_size=len(tasks),
unique_keys=unique_keys,
uppercase=inp.message.upper(),
)
for id, inp in tasks.items()
}
```
#### Typescript
```typescript
export const batchKeyed = hatchet.batchTask({
name: 'batch-keyed',
batch: { maxSize: 2, maxInterval: 200, groupKey: 'input.group' },
fn: async (tasks: Record) => {
const uniqueKeys = new Set(Object.values(tasks).map((i) => i.group)).size;
const batchSize = Object.keys(tasks).length;
const out: Record = {};
Object.entries(tasks).forEach(([id, input]) => {
out[id] = {
batch_key: input.group,
batch_size: batchSize,
unique_keys: uniqueKeys,
uppercase: input.message.toUpperCase(),
};
});
return out;
},
});
```
#### Go
```go
batchKeyed := client.NewStandaloneBatchTask("batch-keyed",
func(ctx hatchet.Context, tasks map[string]KeyedInput) (map[string]KeyedOutput, error) {
uniqueGroups := make(map[string]struct{})
for _, inp := range tasks {
uniqueGroups[inp.Group] = struct{}{}
}
out := make(map[string]KeyedOutput, len(tasks))
for id, inp := range tasks {
out[id] = KeyedOutput{
BatchKey: inp.Group,
BatchSize: len(tasks),
UniqueKeys: len(uniqueGroups),
Uppercase: strings.ToUpper(inp.Message),
}
}
return out, nil
},
hatchet.BatchConfig{
MaxSize: 2,
MaxInterval: durationPtr(200 * time.Millisecond),
GroupKey: stringPtr("input.group"),
},
)
```
#### Ruby
```ruby
BATCH_KEYED = HATCHET.batch_task(
name: 'ruby-e2e-batch-keyed',
batch: Hatchet::BatchTaskConfig.new(max_size: 2, max_interval_ms: 200, group_key: 'input.group')
) do |inputs, _ctx|
unique_keys = inputs.values.map { |i| i['group'] }.uniq.length
batch_size = inputs.length
inputs.transform_values do |input|
{
'batch_key' => input['group'],
'batch_size' => batch_size,
'unique_keys' => unique_keys,
'uppercase' => input['message'].upcase,
}
end
end
```
## Broadcasting outputs
In the default case, batch outputs are mapped 1-1 back to the callers. So if you called a batch task with input `a` and `b` at separate
call sites, you would receive back outputs `a'` and `b'` respectively, despite the fact they would both be executed at the same time, in the same handler.
However, if you wish to return the same input back to all callsites, you can use output broadcasting.
In the following example, every task will receive the same output regardless of call site. Note, however, that this will only return
the same output to every task buffered into the same batch. See the batch flushing rules above for more details.
#### Python
```python
@hatchet.batch_task(
batch_max_size=10,
batch_max_interval=timedelta(seconds=2),
input_validator=SimpleInput,
broadcast_output=True,
)
async def batch_broadcast(
tasks: dict[BatchMemberId, SimpleInput], context: Context
) -> BroadcastOutput:
return BroadcastOutput(sum=sum(len(i.message) for _, i in tasks.items()))
```
#### Typescript
```typescript
export const batchBroadcast = hatchet.batchTask({
name: 'batch-broadcast',
batch: { maxSize: 10, maxInterval: 2_000, broadcastOutput: true },
fn: async (tasks: Record): Promise => {
const sum = Object.values(tasks).reduce((acc, i) => acc + i.message.length, 0);
return { sum };
},
});
```
#### Go
```go
batchBroadcast := client.NewStandaloneBatchTask("batch-broadcast",
func(ctx hatchet.Context, tasks map[string]SimpleInput) (BroadcastSumOutput, error) {
sum := 0
for _, inp := range tasks {
sum += len(inp.Message)
}
return BroadcastSumOutput{Sum: sum}, nil
},
hatchet.BatchConfig{
MaxSize: 10,
MaxInterval: durationPtr(2 * time.Second),
BroadcastOutput: true,
},
)
```
#### Ruby
```ruby
BATCH_BROADCAST = HATCHET.batch_task(
name: 'ruby-e2e-batch-broadcast',
batch: Hatchet::BatchTaskConfig.new(max_size: 10, max_interval_ms: 2_000, broadcast_output: true)
) do |inputs, _ctx|
{ 'sum' => inputs.values.sum { |i| i['message'].length } }
end
```
---
# Durable Tasks
Durable tasks are the fundamental building block of durable execution in Hatchet. A durable task is a task that is comprised of durable execution primitives that conforms to the [core assumptions of durable execution](/v1/durable-execution#core-assumptions). In Hatchet, durable tasks perform two durable operations: they **wait** (for time to pass or events to be received), and they **spawn child tasks**. Every time one of those happens, Hatchet writes a checkpoint to the durable event log. On retries, Hatchet can replay from that checkpoint instead of re-running completed application logic, which gives your tasks exactly-once semantics which you wouldn't get with many other task queue implementations.
Use durable tasks when the shape of work is not known upfront, when parts of the task are hard to make idempotent, or when execution might be interrupted and resumed later. Common examples are agentic loops (often with human-in-the-loop steps), dynamic workflows that choose child workflows at runtime, and long waits that should not hold worker slots. They can also be very simple: sleep, then continue; or wait for an event, then exit.
> **Warning:** Durable tasks must only either call methods on the durable context or spawn
> children, and they must be deterministic given the event history. For example,
> you should _not_ directly access your database or an external API, or generate
> random numbers and use them for control flow inside of a durable task. Have
> your durable task spawn children to do this sort of work instead.
## Determinism in durable tasks
One important thing to keep in mind when writing durable tasks: the code between checkpoints must be deterministic. When a task is evicted and resumed, Hatchet replays the durable event log to rebuild state — it doesn't re-execute completed operations, but it does re-run the code path that led to each checkpoint.
This means a few things in practice:
- **Base decisions on checkpoint outputs**, not on external state that might change between runs (wall-clock time, database reads, random values). If a branch is taken on the first run, it must be taken again on replay.
- **Don't read outside state mid-function** in ways that assume a particular value that may differ between runs.
- **Push side effects into child tasks**. If you need to call external APIs, databases, or other services, do it in child tasks and wait on their results.
If you're ever unsure whether something is safe, ask yourself: "If this task was interrupted and replayed from the last checkpoint, would this code produce the same result?" If yes, you're fine.
## When to use durable tasks
Scenario, Why durable?
**Agentic loops**, Spawn children, collect results, and continue in a loop without losing progress.
**Hard-to-idempotent steps**, Replay from checkpoints instead of re-running already completed logic.
**Runtime-selected workflows**, Choose which child tasks/workflows to run based on intermediate results.
**Long waits / human-in-the-loop**, Wait for timeouts or approvals without holding worker resources.
**Recovering from interruptions**, Resume from checkpoints after worker restarts or crashes.
## How it works
Each time a durable task finishes waiting for something (either a sleep, an event, or a child run to complete), Hatchet checkpoints progress. While the task is waiting, Hatchet can [evict](/v1/task-eviction) it and free the worker slot. When the wait is over, Hatchet re-queues the task, replays the durable event log, and resumes the durable task from the latest checkpoint, as if it had never been evicted.
```mermaid
sequenceDiagram
participant P as Durable Task
participant H as Hatchet
participant W as Workers
P->>H: Spawn Child A, Child B, Child C...N
H-->>P: Evicted (slot freed)
H->>W: Run child workflows
W->>H: Child results
H->>P: Resume from checkpoint
P->>P: Inspect results, decide next step
P->>H: Spawn more children, sleep, or finish
```
This differs from a DAG, where every task and dependency is declared before execution starts. With durable tasks, your code can decide at runtime how many children to spawn, which branch to take, and whether to continue or stop.
## The durable context
When you declare a task as durable, it receives a durable context instead of a regular context. The durable context includes everything in the normal context, plus the durable execution toolkit you'll use to construct your durable task. These methods allow you to [wait for sleeps to complete](/v1/durable-sleep), [wait for events to be received](/v1/durable-event-waits), [wait for children to complete](/v1/child-spawning), or use [or groups](/v1/directed-acyclic-graphs#waiting-on-conditions-with-or-groups) to combine them using boolean "or" logic.
---
# Child Spawning
A task can spawn child tasks at runtime — including other durable tasks or entire DAG workflows. Children run independently on any available worker, and the parent can wait for their results before continuing. You can spawn children out of any task (durable or not).
> **Info:** While child spawning is not unique to durable tasks, we often recommend using
> durable tasks when spawning children is the main responsibility (or one of the
> main responsibilities) of a task. For instance, an agent might spawn many
> children (i.e. tool calls) as its main responsibility, making a durable task a
> good fit.
### Spawning a child task
You can spawn child tasks similarly to how you run tasks normally, but the implementation details differ slightly by language. Any task can spawn child tasks.
#### Python
```python
from examples.fanout.worker import ChildInput, child_wf
# 👀 example: run this inside of a parent task to spawn a child
child_wf.run(
ChildInput(a="b"),
)
```
#### Typescript
```typescript
export const parentSingleChild = hatchet.task({
name: 'parent-single-child',
fn: async () => {
const childRes = await child.run({ N: 1 });
return {
Result: childRes.Value,
};
},
});
```
#### Go
```go
// Inside a parent task
childResult, err := childWorkflow.Run(hCtx, ChildInput{
Value: 1,
})
if err != nil {
return err
}
```
#### Ruby
```ruby
FANOUT_CHILD_WF.run({ "a" => "b" })
```
### Spawning many children at once
You can also spawn children in bulk, exactly the same as you can spawn any other tasks in bulk.
#### Python
```python
async def run_child_workflows(n: int) -> list[dict[str, Any]]:
return await child_wf.aio_run_many(
[
child_wf.create_bulk_run_item(
input=ChildInput(a=str(i)),
)
for i in range(n)
]
)
```
#### Typescript
```typescript
type ParentInput = {
N: number;
};
export const parent = hatchet.task({
name: 'parent',
fn: async (input: ParentInput, ctx) => {
const n = input.N;
const promises = [];
for (let i = 0; i < n; i++) {
promises.push(child.run({ N: i }));
}
const childRes = await Promise.all(promises);
const sum = childRes.reduce((acc, curr) => acc + curr.Value, 0);
return {
Result: sum,
};
},
});
```
#### Go
```go
// Run multiple child tasks in parallel using goroutines
var wg sync.WaitGroup
var mu sync.Mutex
results := make([]*ChildOutput, 0, n)
wg.Add(n)
for i := 0; i < n; i++ {
go func(index int) {
defer wg.Done()
result, err := childWorkflow.Run(hCtx, ChildInput{Value: index})
if err != nil {
return
}
var childOutput ChildOutput
err = result.Into(&childOutput)
if err != nil {
return
}
mu.Lock()
results = append(results, &childOutput)
mu.Unlock()
}(i)
}
wg.Wait()
```
#### Ruby
```ruby
def run_child_workflows(n)
FANOUT_CHILD_WF.run_many(
n.times.map do |i|
FANOUT_CHILD_WF.create_bulk_run_item(
input: { "a" => i.to_s }
)
end
)
end
```
### What you can spawn
A durable task can spawn any runnable:
Child type, Example
**Regular task**, Spawn a stateless task for a quick computation or API call.
**Durable task**, Spawn another durable task that has its own checkpoints, sleeps, and event waits.
**DAG workflow**, Spawn an entire multi-task workflow and wait for its final output.
### Error handling
#### Python
```python
try:
child_wf.run(
ChildInput(a="b"),
)
except Exception as e:
print(f"Child workflow failed: {e}")
```
#### Typescript
```typescript
export const withErrorHandling = hatchet.task({
name: 'parent-error-handling',
fn: async () => {
try {
const childRes = await child.run({ N: 1 });
return {
Result: childRes.Value,
};
} catch (error) {
// decide how to proceed here
return {
Result: -1,
};
}
},
});
```
#### Go
```go
result, err := childWorkflow.Run(hCtx, ChildInput{Value: 1})
if err != nil {
// Handle error from child workflow
fmt.Printf("Child workflow failed: %v\n", err)
// Decide how to proceed - retry, skip, or fail the parent
}
```
#### Ruby
```ruby
begin
FANOUT_CHILD_WF.run({ "a" => "b" })
rescue StandardError => e
puts "Child workflow failed: #{e.message}"
end
```
## Ways to use child spawning
### Fan-out: spawning many children in parallel
Process a list of items whose length is only known at runtime. Spawn one child per item, collect all results, then continue. Document processing and batch processing are the canonical examples: when a batch of files arrives, a parent fans out to one child per document; each child parses, extracts, and validates its document in parallel across your worker fleet.
[Concurrency](/v1/concurrency) controls how many children run at once. Hatchet distributes child tasks across available workers, so adding workers increases throughput without code changes. For rate-limited external services (OCR, LLM APIs), combine with [Rate Limits](/v1/rate-limits) to throttle child execution across all workers.
### Agent reasoning loops
An agent loop runs by having a durable task spawn a new child run of itself with updated input until a termination condition is met. Each iteration is a separate child task, so you get full observability in the dashboard. AI agents use this when they reason about what to do next, spawn a subtask (or a sub-workflow), inspect the result, and decide whether to continue, branch, or stop.
### Spawning trees of work
A durable task can spawn child durable tasks, each of which may spawn their own children. This creates a tree of work that's entirely driven by runtime logic — useful for crawlers, recursive search, and tree-structured computations.
```go
// Inside a parent task
childResult, err := childWorkflow.Run(hCtx, ChildInput{
Value: 1,
})
if err != nil {
return err
}
```
---
# Durable Sleep
A durable task can elect to **sleep** for either a specified period of time or until a provided time, which pauses task execution. While the task is sleeping, no resources are consumed, and the task can also be [evicted](/v1/task-eviction) in order to free the worker slot.
Unlike a language-level sleep (e.g. `time.sleep` in Python or `setTimeout` in Node), durable sleep is guaranteed to respect the original duration across interruptions of the durable task, worker crashes, and so on. A language-level sleep ties the wait to the local process, so if the process restarts, the sleep starts over from scratch.
### Using durable sleep
The `DurableContext` exposes helper methods to allow you to sleep for either a specific duration of time, or until a certain time.
#### Python
```python
@hatchet.durable_task(name="DurableSleepTask")
async def durable_sleep_task(input: EmptyModel, ctx: DurableContext) -> None:
res = await ctx.aio_sleep_for(timedelta(seconds=5))
print("got result", res)
```
#### Typescript
```typescript
durableSleep.durableTask({
name: 'durable-sleep',
executionTimeout: '10m',
fn: async (_, ctx) => {
ctx.logger.info('sleeping for 5s');
const sleepRes = await ctx.sleepFor('5s');
ctx.logger.info('done sleeping for 5s', { sleepRes });
return {
Value: 'done',
};
},
});
```
#### Go
```go
task := client.NewStandaloneDurableTask("long-running-task", func(ctx hatchet.DurableContext, input DurableInput) (DurableOutput, error) {
log.Printf("Starting task, will sleep for %d seconds", input.Delay)
if _, err := ctx.SleepFor(time.Duration(input.Delay) * time.Second); err != nil {
return DurableOutput{}, err
}
log.Printf("Finished sleeping, processing message: %s", input.Message)
return DurableOutput{
ProcessedAt: time.Now().Format(time.RFC3339),
Message: "Processed: " + input.Message,
}, nil
})
```
#### Ruby
```ruby
DURABLE_SLEEP_TASK = HATCHET.durable_task(name: "DurableSleepTask") do |input, ctx|
res = ctx.sleep_for(duration: 5)
puts "got result #{res}"
end
```
---
# Durable Event Waits
Tasks can pause until an external event arrives before continuing. This is the foundation for human-in-the-loop workflows, webhook-driven pipelines, and any process that depends on signals from outside the task.
Events are delivered by pushing events into Hatchet either [using one of the SDKs](/v1/events#pushing-events-to-hatchet) or via an [incoming webhook](/v1/webhooks). The event key you push must match the key your task is waiting for.
Waiting for an event lets a durable task pause until an event arrives. Even if the task is interrupted and requeued while waiting, the event will still be processed. When it resumes, it reads the event from the durable event log and continues.
## Establishing an event wait
Your durable tasks can establish an event wait by using helper methods on the `DurableContext`.
#### Python
```python
@hatchet.durable_task(name="DurableEventTask")
async def durable_event_task(input: EmptyModel, ctx: DurableContext) -> None:
res = await ctx.aio_wait_for_event(
"user:update",
)
print("got event", res)
```
#### Typescript
```typescript
export const durableEvent = hatchet.durableTask({
name: 'durable-event',
executionTimeout: '10m',
fn: async (_, ctx) => {
const res = await ctx.waitForEvent(EVENT_KEY);
ctx.logger.info('res', { res });
return {
Value: 'done',
};
},
});
```
#### Go
```go
task := client.NewStandaloneDurableTask("durable-event-task", func(ctx hatchet.DurableContext, input DurableInput) (DurableOutput, error) {
log.Printf("Waiting for user:update event, message: %s", input.Message)
if _, err := ctx.WaitForEvent("user:update", ""); err != nil {
return DurableOutput{}, err
}
log.Printf("Got event, processing message: %s", input.Message)
return DurableOutput{
ProcessedAt: time.Now().Format(time.RFC3339),
Message: "Processed: " + input.Message,
}, nil
})
```
#### Ruby
```ruby
DURABLE_EVENT_TASK = HATCHET.durable_task(name: "DurableEventTask") do |input, ctx|
res = ctx.wait_for(
"event",
Hatchet::UserEventCondition.new(event_key: "user:update")
)
puts "got event #{res}"
end
DURABLE_EVENT_TASK_WITH_FILTER = HATCHET.durable_task(name: "DurableEventWithFilterTask") do |input, ctx|
```
## Matching CEL expressions
In general, you'll want to match the payload of the incoming event to some data that your task knows about to make sure the event you receive is the _right_ one. For instance, you might want to make sure that the event payload contains the correct user or organization id in order to consider the event wait as having been completed. To do this, you can provide a [CEL expression](https://celbyexample.com/) in addition to the event key when establishing the wait.
#### Python
```python
res = await ctx.aio_wait_for_event("user:update", "input.user_id == '1234'")
```
#### Typescript
```typescript
const res = await ctx.waitForEvent(EVENT_KEY, "input.userId == '1234'");
```
#### Go
```go
if _, err := ctx.WaitForEvent("user:update", "input.user_id == '1234'"); err != nil {
return DurableOutput{}, err
}
```
#### Ruby
```ruby
res = ctx.wait_for(
"event",
Hatchet::UserEventCondition.new(
event_key: "user:update",
expression: "input.user_id == '1234'"
)
)
puts "got event #{res}"
end
```
## Lookback Windows
Event waits can also optionally look back in time for recent events that have come in, which is often useful for preventing race conditions. An important caveat is that in order look up previous events, the wait must also be established alongside a **scope**, which must also be pushed with the event itself. The scope serves as a hint to help Hatchet narrow down the pool of candidate events.
As a simple example, if you're establishing an event wait with a CEL expression like `input.user_id == 1234`, then you know that `{"user_id": 1234}` is present on the event payload, so you might use the string `user_id:1234` as the scope, **both when establishing the wait and pushing the event.**
#### Python
```python
event = await ctx.aio_wait_for_event(
key="user:create",
expression=f"input.user_id == {input.user_id}",
scope=f"user_id:{input.user_id}",
lookback_window=timedelta(minutes=1),
payload_validator=LookbackEventPayload,
)
```
#### Typescript
```typescript
const event = await ctx.waitForEvent(
'user:create',
`input.user_id == ${input.userId}`,
lookbackEventPayloadSchema,
`user_id:${input.userId}`,
'1m'
);
```
#### Go
> **Info:** Lookback window support for the Go SDK is coming soon. Join our
> [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
> **Info:** Lookback window support for the Ruby SDK is coming soon. Join our
> [Discord](https://hatchet.run/discord) to stay up to date.
---
# Task Eviction
Since durable tasks are often waiting for sleeps, events, or child runs, they spend much of their time inactive. When this happens, the durable task no longer needs to hold a slot on the worker until the thing it's waiting for completes. Hatchet has the option to **evict** durable tasks from workers when they're in one of these waiting states, in order to free up slots on the worker. This allows the worker to pick up additional durable tasks, and then resume the original one from where it left off when the wait is satisfied, without keeping the idle task around.
> **Info:** It's important to note that eviction is a fundamental difference between
> durable and non-durable tasks. Only durable tasks can be evicted from and
> restored on workers, as opposed to non-durable tasks, which will hold slots
> throughout the entirety of their execution, even while they're inactive.
### Configuring Eviction Policies
Every durable task can configure an **eviction policy**, which tells Hatchet how to respond when that task hits a wait. There are a few configuration options available to you to choose from:
1. You can set a TTL, which tells Hatchet that the durable task should only be evicted after some amount of uninterrupted time has been spent waiting.
2. You can enable or disable **capacity-based eviction**, which allows Hatchet to optimistically evict the durable task if the worker is running out of slots and wants to pick up a new durable task run, but would be unable to if the waiting task could not be evicted.
3. Finally, you can provide a **priority**, which tells Hatchet what order candidate durable tasks should be evicted in (lower priority values will be evicted first when choosing between multiple candidate durable tasks to evict).
Start by declaring a policy:
#### Python
```python
EVICTION_POLICY = EvictionPolicy(
ttl=timedelta(seconds=EVICTION_TTL_SECONDS),
allow_capacity_eviction=True,
priority=0,
)
```
#### Typescript
```typescript
const EVICTION_POLICY: EvictionPolicy = {
ttl: `${EVICTION_TTL_SECONDS}s`,
allowCapacityEviction: true,
priority: 0,
};
```
#### Go
```go
evictionPolicy := &hatchet.EvictionPolicy{
TTL: evictionTTLSeconds * time.Second,
AllowCapacityEviction: true,
Priority: 0,
}
```
#### Ruby
```ruby
EVICTION_POLICY = Hatchet::EvictionPolicy.new(
ttl: EVICTION_TTL_SECONDS,
allow_capacity_eviction: true,
priority: 0,
)
```
Then attach it to a durable task. Any time the task enters a wait (sleep, event wait, child spawn) the policy's TTL and capacity-eviction settings are honored:
#### Python
```python
@hatchet.durable_task(
execution_timeout=timedelta(minutes=5),
eviction_policy=EVICTION_POLICY,
)
async def evictable_sleep(input: EmptyModel, ctx: DurableContext) -> dict[str, Any]:
"""Sleeps long enough for the TTL-based eviction to kick in."""
await ctx.aio_sleep_for(timedelta(seconds=LONG_SLEEP_SECONDS))
return {"status": "completed"}
```
#### Typescript
```typescript
export const evictableSleep = hatchet.durableTask({
name: 'evictable-sleep',
executionTimeout: '5m',
evictionPolicy: EVICTION_POLICY,
fn: async (_input, ctx) => {
await ctx.sleepFor(`${LONG_SLEEP_SECONDS}s`);
return { status: 'completed' };
},
});
```
#### Go
```go
evictableSleep := client.NewStandaloneDurableTask("evictable-sleep",
func(ctx hatchet.DurableContext, input EmptyInput) (EvictionOutput, error) {
if _, err := ctx.SleepFor(longSleepSeconds * time.Second); err != nil {
return EvictionOutput{}, err
}
return EvictionOutput{Status: "completed"}, nil
},
hatchet.WithExecutionTimeout(5*time.Minute),
hatchet.WithEvictionPolicy(evictionPolicy),
)
```
#### Ruby
```ruby
EVICTABLE_SLEEP = HATCHET.durable_task(
name: "evictable_sleep",
execution_timeout: 300,
eviction_policy: EVICTION_POLICY,
) do |_input, ctx|
ctx.sleep_for(duration: LONG_SLEEP_SECONDS)
{ "status" => "completed" }
end
```
To opt a durable task out of eviction entirely, set `allowCapacityEviction` to `false` and leave the TTL unset. The task will hold its slot through waits:
#### Python
```python
@hatchet.durable_task(
execution_timeout=timedelta(minutes=5),
eviction_policy=EvictionPolicy(
ttl=None,
allow_capacity_eviction=False,
priority=0,
),
)
async def non_evictable_sleep(input: EmptyModel, ctx: DurableContext) -> dict[str, Any]:
"""Has eviction disabled -- should never be evicted."""
await ctx.aio_sleep_for(timedelta(seconds=30))
return {"status": "completed"}
```
#### Typescript
```typescript
export const nonEvictableSleep = hatchet.durableTask({
name: 'non-evictable-sleep',
executionTimeout: '5m',
evictionPolicy: {
ttl: undefined,
allowCapacityEviction: false,
priority: 0,
},
fn: async (_input, ctx) => {
await ctx.sleepFor('10s');
return { status: 'completed' };
},
});
```
#### Go
```go
nonEvictablePolicy := &hatchet.EvictionPolicy{
AllowCapacityEviction: false,
Priority: 0,
}
nonEvictableSleep := client.NewStandaloneDurableTask("non-evictable-sleep",
func(ctx hatchet.DurableContext, input EmptyInput) (EvictionOutput, error) {
if _, err := ctx.SleepFor(10 * time.Second); err != nil {
return EvictionOutput{}, err
}
return EvictionOutput{Status: "completed"}, nil
},
hatchet.WithExecutionTimeout(5*time.Minute),
hatchet.WithEvictionPolicy(nonEvictablePolicy),
)
```
#### Ruby
```ruby
NON_EVICTABLE_SLEEP = HATCHET.durable_task(
name: "non_evictable_sleep",
execution_timeout: 300,
eviction_policy: NON_EVICTABLE_POLICY,
) do |_input, ctx|
ctx.sleep_for(duration: 10)
{ "status" => "completed" }
end
```
### Task Resumption
Hatchet will **resume** a task in an evicted state when the thing it was waiting for is satisfied. When this happens, the durable task is re-triggered on the worker, and its event log is replayed up to the checkpoint where it left off, at which point it continues making progress.
> **Warning:** Since durable tasks can be evicted or resumed, it's important to make sure
> that a durable task doesn't perform any operations that you wouldn't want to
> occur multiple times. For instance, non-idempotent writes to a database,
> sending emails, or computationally expensive work are generally not good
> candidates for the kinds of things to do in a durable task, since that logic
> will be rerun every time the task is resumed.
---
# DAGs as Durable Workflows
A directed acyclic graph (DAG) is a workflow where every task, along with the dependencies between them, is declared upfront, before the workflow runs. At runtime, Hatchet schedules the tasks in the right order, runs tasks that don't depend on each other in parallel, and passes the outputs of parent tasks to their children automatically.
DAGs in Hatchet are a form of [durable execution](/v1/durable-execution) by definition. Every time a task in a DAG completes, Hatchet persists its status and result so that the DAG can be retried without re-executing succeeded parts. This gives DAGs similar exactly-once-style guarantees you'd get from a [durable task](/v1/durable-tasks), without having to write any explicit durable execution code yourself.
On top of those durable properties, DAGs come bundled with a set of workflow-building features you'd otherwise have to construct by hand: automatic parallelism between independent tasks, typed inputs and outputs flowing from parents to children, [branching](#branching-with-parent-conditions), [or groups](#waiting-on-conditions-with-or-groups) for expressing complex wait conditions, and a clear visual representation of the workflow in the Hatchet dashboard.
Common examples of good fits for DAGs are ETL pipelines such as for document or image processing, and CI/CD-style workflows.
## Defining a workflow
To define a DAG, start by declaring a workflow.
#### Python
```python
dag_workflow = hatchet.workflow(name="DAGWorkflow")
```
#### Typescript
```typescript
// First, we declare the workflow
export const dag = hatchet.workflow({
name: 'simple',
});
```
#### Go
```go
workflow := client.NewWorkflow("dag-workflow")
```
#### Ruby
```ruby
DAG_WORKFLOW = HATCHET.workflow(name: "DAGWorkflow")
```
### Defining a task
Once you have a workflow, you can add tasks to it. Each task is a function that receives the workflow's input and optionally returns an output. Just like a standalone task, every task in a DAG can declare its own retries, timeouts, concurrency settings, and so on. For more on how tasks work, see the [tasks documentation](/v1/tasks).
#### Python
```python
@dag_workflow.task(execution_timeout=timedelta(seconds=5))
def step1(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
```
#### Typescript
```typescript
// Next, we declare the tasks bound to the workflow
const toLower = dag.task({
name: 'to-lower',
fn: (input) => {
return {
TransformedMessage: input.Message.toLowerCase(),
};
},
});
```
#### Go
```go
step1 := workflow.NewTask("step-1", func(ctx hatchet.Context, input Input) (StepOutput, error) {
return StepOutput{
Step: 1,
Result: input.Value * 2,
}, nil
})
```
#### Ruby
```ruby
STEP1 = DAG_WORKFLOW.task(:step1, execution_timeout: 5) do |input, ctx|
{ "random_number" => rand(1..100) }
end
STEP2 = DAG_WORKFLOW.task(:step2, execution_timeout: 5) do |input, ctx|
{ "random_number" => rand(1..100) }
end
```
### Adding task dependencies
Once you have more than one task on a workflow, you can start to declare dependencies between them. A task can declare one or more **parent** tasks, meaning that those parent tasks must complete successfully before the child task is allowed to run. Tasks that don't depend on each other will be run in parallel. When a task runs, the outputs of its parents are available on its context object, so data flows naturally from one part of the DAG to the next.
#### Python
```python
@dag_workflow.task(execution_timeout=timedelta(seconds=5))
async def step2(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
@dag_workflow.task(parents=[step1, step2])
async def step3(input: EmptyModel, ctx: Context) -> RandomSum:
one = ctx.task_output(step1).random_number
two = ctx.task_output(step2).random_number
return RandomSum(sum=one + two)
```
#### Typescript
```typescript
dag.task({
name: 'reverse',
parents: [toLower],
fn: async (input, ctx) => {
const lower = await ctx.parentOutput(toLower);
return {
Original: input.Message,
Transformed: lower.TransformedMessage.split('').reverse().join(''),
};
},
});
```
#### Go
```go
step2 := workflow.NewTask("step-2", func(ctx hatchet.Context, input Input) (StepOutput, error) {
// Get output from step 1
var step1Output StepOutput
if err := ctx.ParentOutput(step1, &step1Output); err != nil {
return StepOutput{}, err
}
return StepOutput{
Step: 2,
Result: step1Output.Result + 10,
}, nil
}, hatchet.WithParents(step1))
```
#### Ruby
```ruby
DAG_WORKFLOW.task(:step3, parents: [STEP1, STEP2]) do |input, ctx|
one = ctx.task_output(STEP1)["random_number"]
two = ctx.task_output(STEP2)["random_number"]
{ "sum" => one + two }
end
DAG_WORKFLOW.task(:step4, parents: [STEP1, :step3]) do |input, ctx|
puts(
"executed step4",
Time.now.strftime("%H:%M:%S"),
input.inspect,
ctx.task_output(STEP1).inspect,
ctx.task_output(:step3).inspect
)
{ "step4" => "step4" }
end
```
## Running a workflow
Once you've defined a workflow and registered it on a [worker](/v1/workers), you can trigger it in all of the same ways you can trigger a standalone task. You can run it and wait for the result, enqueue it and let it run in the background, schedule it for the future, and so on. See the [Running Tasks](/v1/running-your-task) documentation for the full set of options.
## Branching with parent conditions
Even though the structure of a DAG is fixed when you define it, individual tasks can still decide at runtime whether or not they should run, based on data produced earlier in the workflow. **Parent conditions** let a task inspect the output of one of its parents and either skip itself or cancel itself based on what that output contains. There are two operators available:
- **`skip_if`**: Skip this task if the parent's output matches the condition. Downstream tasks can [check whether a parent was skipped](#checking-if-a-parent-was-skipped) and behave accordingly.
- **`cancel_if`**: Cancel this task if the parent's output matches the condition.
> **Warning:** A task cancelled by `cancel_if` behaves like any other cancellation in
> Hatchet, meaning its downstream dependents will be cancelled as well.
A common way to use parent conditions is to express sibling branches in a DAG. You declare a base task that returns some data, and then add two sibling tasks with complementary `skip_if` conditions, so that exactly one of them runs on any given workflow execution.
First, declare the base task that returns the value the branches will key off of:
#### Python
```python
@task_condition_workflow.task()
def start(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
```
#### Typescript
```typescript
const start = taskConditionWorkflow.task({
name: 'start',
fn: () => {
return {
randomNumber: Math.floor(Math.random() * 100) + 1,
};
},
});
```
#### Go
```go
start := workflow.NewTask("start", func(ctx hatchet.Context, _ any) (StepOutput, error) {
return StepOutput{RandomNumber: rand.Intn(100) + 1}, nil //nolint:gosec
})
```
#### Ruby
```ruby
COND_START = TASK_CONDITION_WORKFLOW.task(:start) do |input, ctx|
{ "random_number" => rand(1..100) }
end
```
Then add two branches that each use a `skip_if` parent condition to decide whether they should run, based on the output of the base task:
#### Python
```python
@task_condition_workflow.task(
parents=[wait_for_sleep],
skip_if=[
ParentCondition(
parent=wait_for_sleep,
expression="output.random_number > 50",
)
],
)
def left_branch(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
@task_condition_workflow.task(
parents=[wait_for_sleep],
skip_if=[
ParentCondition(
parent=wait_for_sleep,
expression="output.random_number <= 50",
)
],
)
def right_branch(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
```
#### Typescript
```typescript
const leftBranch = taskConditionWorkflow.task({
name: 'leftBranch',
parents: [waitForSleep],
skipIf: [new ParentCondition(waitForSleep, 'output.randomNumber > 50')],
fn: () => {
return {
randomNumber: Math.floor(Math.random() * 100) + 1,
};
},
});
const rightBranch = taskConditionWorkflow.task({
name: 'rightBranch',
parents: [waitForSleep],
skipIf: [new ParentCondition(waitForSleep, 'output.randomNumber <= 50')],
fn: () => {
return {
randomNumber: Math.floor(Math.random() * 100) + 1,
};
},
});
```
#### Go
```go
leftBranch := workflow.NewTask("left-branch", func(ctx hatchet.Context, _ any) (StepOutput, error) {
return StepOutput{RandomNumber: rand.Intn(100) + 1}, nil //nolint:gosec
},
hatchet.WithParents(waitForSleep),
hatchet.WithSkipIf(hatchet.ParentCondition(waitForSleep, "output.random_number > 50")),
)
rightBranch := workflow.NewTask("right-branch", func(ctx hatchet.Context, _ any) (StepOutput, error) {
return StepOutput{RandomNumber: rand.Intn(100) + 1}, nil //nolint:gosec
},
hatchet.WithParents(waitForSleep),
hatchet.WithSkipIf(hatchet.ParentCondition(waitForSleep, "output.random_number <= 50")),
)
```
#### Ruby
```ruby
LEFT_BRANCH = TASK_CONDITION_WORKFLOW.task(
:left_branch,
parents: [WAIT_FOR_SLEEP],
skip_if: [
Hatchet::ParentCondition.new(
parent: WAIT_FOR_SLEEP,
expression: "output.random_number > 50"
)
]
) do |input, ctx|
{ "random_number" => rand(1..100) }
end
RIGHT_BRANCH = TASK_CONDITION_WORKFLOW.task(
:right_branch,
parents: [WAIT_FOR_SLEEP],
skip_if: [
Hatchet::ParentCondition.new(
parent: WAIT_FOR_SLEEP,
expression: "output.random_number <= 50"
)
]
) do |input, ctx|
{ "random_number" => rand(1..100) }
end
```
### Checking if a parent was skipped
When two sibling branches both feed into a common downstream task, the downstream task often needs to know which of its parents actually ran, as opposed to which one was skipped. You can check this on the context with `ctx.was_skipped`:
#### Python
```python
@task_condition_workflow.task(
parents=[
start,
wait_for_sleep,
wait_for_event,
skip_on_event,
left_branch,
right_branch,
],
)
def sum(input: EmptyModel, ctx: Context) -> RandomSum:
one = ctx.task_output(start).random_number
two = ctx.task_output(wait_for_event).random_number
three = ctx.task_output(wait_for_sleep).random_number
four = (
ctx.task_output(skip_on_event).random_number
if not ctx.was_skipped(skip_on_event)
else 0
)
five = (
ctx.task_output(left_branch).random_number
if not ctx.was_skipped(left_branch)
else 0
)
six = (
ctx.task_output(right_branch).random_number
if not ctx.was_skipped(right_branch)
else 0
)
return RandomSum(sum=one + two + three + four + five + six)
```
#### Typescript
```typescript
taskConditionWorkflow.task({
name: 'sum',
parents: [start, waitForSleep, waitForEvent, skipOnEvent, leftBranch, rightBranch],
fn: async (_, ctx: Context) => {
const one = (await ctx.parentOutput(start)).randomNumber;
const two = (await ctx.parentOutput(waitForEvent)).randomNumber;
const three = (await ctx.parentOutput(waitForSleep)).randomNumber;
const four = (await ctx.parentOutput(skipOnEvent))?.randomNumber || 0;
const five = (await ctx.parentOutput(leftBranch))?.randomNumber || 0;
const six = (await ctx.parentOutput(rightBranch))?.randomNumber || 0;
return {
sum: one + two + three + four + five + six,
};
},
});
```
#### Go
```go
_ = workflow.NewTask("sum", func(ctx hatchet.Context, _ any) (RandomSum, error) {
var startOut StepOutput
err := ctx.ParentOutput(start, &startOut)
if err != nil {
return RandomSum{}, err
}
var waitForEventOut StepOutput
err = ctx.ParentOutput(waitForEvent, &waitForEventOut)
if err != nil {
return RandomSum{}, err
}
var waitForSleepOut StepOutput
err = ctx.ParentOutput(waitForSleep, &waitForSleepOut)
if err != nil {
return RandomSum{}, err
}
total := startOut.RandomNumber + waitForEventOut.RandomNumber + waitForSleepOut.RandomNumber
if !ctx.WasSkipped(skipOnEvent) {
var out StepOutput
err = ctx.ParentOutput(skipOnEvent, &out)
if err == nil {
total += out.RandomNumber
}
}
if !ctx.WasSkipped(leftBranch) {
var out StepOutput
err = ctx.ParentOutput(leftBranch, &out)
if err == nil {
total += out.RandomNumber
}
}
if !ctx.WasSkipped(rightBranch) {
var out StepOutput
err = ctx.ParentOutput(rightBranch, &out)
if err == nil {
total += out.RandomNumber
}
}
return RandomSum{Sum: total}, nil
}, hatchet.WithParents(
start,
waitForSleep,
waitForEvent,
skipOnEvent,
leftBranch,
rightBranch,
))
```
#### Ruby
```ruby
TASK_CONDITION_WORKFLOW.task(
:sum,
parents: [COND_START, WAIT_FOR_SLEEP, WAIT_FOR_EVENT, SKIP_ON_EVENT, LEFT_BRANCH, RIGHT_BRANCH]
) do |input, ctx|
one = ctx.task_output(COND_START)["random_number"]
two = ctx.task_output(WAIT_FOR_EVENT)["random_number"]
three = ctx.task_output(WAIT_FOR_SLEEP)["random_number"]
four = ctx.was_skipped?(SKIP_ON_EVENT) ? 0 : ctx.task_output(SKIP_ON_EVENT)["random_number"]
five = ctx.was_skipped?(LEFT_BRANCH) ? 0 : ctx.task_output(LEFT_BRANCH)["random_number"]
six = ctx.was_skipped?(RIGHT_BRANCH) ? 0 : ctx.task_output(RIGHT_BRANCH)["random_number"]
{ "sum" => one + two + three + four + five + six }
end
```
## Waiting on conditions with or groups
In addition to parent conditions, a task in a DAG can wait for external signals before it starts running. It might wait for a sleep timer to expire, for a [user event](/v1/events) to arrive, or for some combination of these alongside parent conditions. You compose conditions like these using **or groups**.
An or group is a set of conditions combined with an `Or` operator, which evaluates to `True` as soon as **at least one** of the conditions inside it is satisfied. If you declare more than one or group on the same task, the groups are combined with `AND`, meaning that every group must have at least one of its conditions satisfied before the task is allowed to run. Between these two operators, you can express arbitrarily complex wait conditions in [conjunctive normal form](https://en.wikipedia.org/wiki/Conjunctive_normal_form) (CNF).
### Sleep + event example
A common pattern is to combine a sleep condition and an event condition in the same or group, so that the task proceeds as soon as either an external signal arrives _or_ a timeout expires, whichever happens first. This is a natural fit for human-in-the-loop workflows, where you want to put a deadline on how long you'll wait for a response before moving on.
#### Python
```python
@task_condition_workflow.task(
parents=[start],
wait_for=[
or_(
SleepCondition(duration=timedelta(minutes=1)),
UserEventCondition(event_key="wait_for_event:start"),
)
],
)
def wait_for_event(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
```
#### Typescript
```typescript
const waitForEvent = taskConditionWorkflow.task({
name: 'waitForEvent',
parents: [start],
waitFor: [Or(new SleepCondition('1m'), new UserEventCondition('wait_for_event:start', 'true'))],
fn: () => {
return {
randomNumber: Math.floor(Math.random() * 100) + 1,
};
},
});
```
#### Go
```go
waitForEvent := workflow.NewTask("wait-for-event", func(ctx hatchet.Context, _ any) (StepOutput, error) {
return StepOutput{RandomNumber: rand.Intn(100) + 1}, nil //nolint:gosec
},
hatchet.WithParents(start),
hatchet.WithWaitFor(hatchet.OrCondition(
hatchet.SleepCondition(1*time.Minute),
hatchet.UserEventCondition("wait_for_event:start", ""),
)),
)
```
#### Ruby
```ruby
WAIT_FOR_EVENT = TASK_CONDITION_WORKFLOW.task(
:wait_for_event,
parents: [COND_START],
wait_for: [
Hatchet.or_(
Hatchet::SleepCondition.new(60),
Hatchet::UserEventCondition.new(event_key: "wait_for_event:start")
)
]
) do |input, ctx|
{ "random_number" => rand(1..100) }
end
```
### Combining multiple or groups
For more complicated wait logic, you can declare more than one or group on the same task. As an example, consider these three conditions:
- **Condition A**: A parent task's output is greater than 50.
- **Condition B**: A 30 second sleep timer expires.
- **Condition C**: A `payment:processed` event arrives.
If you want the task to proceed when `(A or B)` **and** `(A or C)` are both satisfied, you'd declare two separate or groups on the task: one containing `A or B`, and the other containing `A or C`. The task will only start once both groups have been satisfied. If `A` is true, both groups pass immediately. If `A` is false, the task needs both `B` (the sleep expires) and `C` (the event arrives) before it can run.
#### Python
```python
@task_condition_workflow.task(
parents=[start],
wait_for=[
or_(
SleepCondition(duration=timedelta(seconds=30), readable_data_key="first"),
),
or_(
SleepCondition(duration=timedelta(seconds=30), readable_data_key="second"),
UserEventCondition(event_key="payment:processed"),
),
],
)
def wait_for_or_groups(input: EmptyModel, ctx: Context) -> StepOutput:
return StepOutput(random_number=random.randint(1, 100))
```
#### Typescript
```typescript
taskConditionWorkflow.task({
name: 'waitForOrGroups',
parents: [start],
waitFor: [
Or(new ParentCondition(start, 'output.randomNumber > 50'), new SleepCondition('30s')),
Or(
new ParentCondition(start, 'output.randomNumber > 50'),
new UserEventCondition('payment:processed', 'true')
),
],
fn: () => {
return {
randomNumber: Math.floor(Math.random() * 100) + 1,
};
},
});
```
---
# Dockerizing Hatchet Applications
This guide explains how to create Dockerfiles for Hatchet applications. There are examples for Python, TypeScript, Go, and Ruby applications here.
## Entrypoint Configuration for Hatchet
Before creating your Dockerfile, understand that Hatchet workers require specific entry point configuration:
1. The entry point must run code that runs the Hatchet worker. This can be done by calling the `worker.start()` method in your respective SDK.
2. Proper environment variables must be set for Hatchet SDK
3. The worker should be configured to handle your workflows using the `worker.register` method or by passing workflows into the worker constructor or factory.
## Example Dockerfiles
#### Python - Poetry
```dockerfile
FROM python:3.13-slim
ENV PYTHONUNBUFFERED=1 \
POETRY_VERSION=1.4.2 \
HATCHET_ENV=production
# Install system dependencies and Poetry
RUN apt-get update && \
apt-get install -y curl && \
curl -sSL https://install.python-poetry.org | python3 - && \
ln -s /root/.local/bin/poetry /usr/local/bin/poetry && \
apt-get clean && \
rm -rf /var/lib/apt/lists/\*
WORKDIR /app
COPY pyproject.toml poetry.lock\* /app/
RUN poetry config virtualenvs.create false && \
poetry install --no-interaction --no-ansi
COPY . /app
CMD ["poetry", "run", "python", "worker.py"]
```
> **Info:** If you're using a poetry script to run your worker, you can replace `poetry run python worker.py` with `poetry run ` in the CMD.
#### Python - pip
```dockerfile
FROM python:3.13-slim
ENV PYTHONUNBUFFERED=1 \
HATCHET_ENV=production
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app
CMD ["python", "worker.py"]
```
#### JavaScript - npm
```dockerfile
# Stage 1: Build
FROM node:18 AS builder
WORKDIR /app
COPY package\*.json ./
RUN npm ci
COPY . .
RUN npm run build
# Stage 2: Production
FROM node:22-alpine
WORKDIR /app
COPY package\*.json ./
RUN npm ci --omit=dev
COPY --from=builder /app/dist ./dist
ENV NODE_ENV=production
CMD ["node", "dist/worker.js"]
```
> **Info:** Use `npm ci` instead of `npm install` for more reliable builds. It's faster and ensures consistent installs across environments.
#### JavaScript - pnpm
```dockerfile
# Stage 1: Build
FROM node:18 AS builder
WORKDIR /app
# Install pnpm
RUN npm install -g pnpm
COPY pnpm-lock.yaml package.json ./
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm build
# Stage 2: Production
FROM node:22-alpine
WORKDIR /app
RUN npm install -g pnpm
COPY pnpm-lock.yaml package.json ./
RUN pnpm install --frozen-lockfile --prod
COPY --from=builder /app/dist ./dist
ENV NODE_ENV=production
CMD ["node", "dist/worker.js"]
```
> **Info:** PNPM's `--frozen-lockfile` flag ensures consistent installs and fails if an update is needed.
#### JavaScript - yarn
```dockerfile
# Stage 1: Build
FROM node:18 AS builder
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build
# Stage 2: Production
FROM node:22-alpine
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile --production
COPY --from=builder /app/dist ./dist
ENV NODE_ENV=production
CMD ["node", "dist/worker.js"]
```
> **Info:** Yarn's `--frozen-lockfile` ensures your dependencies match the lock file exactly.
#### Go
```dockerfile
# Stage 1: Build
FROM golang:1.26-alpine3.21 AS builder
WORKDIR /app
COPY . .
RUN go mod download
RUN go build -o hatchet-worker .
# Stage 2: Production
FROM golang:1.26-alpine3.21
WORKDIR /app
COPY --from=builder hatchet-worker .
CMD ["/app/hatchet-worker"]
```
#### Ruby
```dockerfile
FROM ruby:3.3-slim
ENV HATCHET_ENV=production
# Install system dependencies for native gems
RUN apt-get update && \
apt-get install -y build-essential && \
apt-get clean && \
rm -rf /var/lib/apt/lists/\*
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN bundle config set --local without 'development test' && \
bundle install
COPY . /app
CMD ["bundle", "exec", "ruby", "worker.rb"]
```
> **Info:** If you're using a Rake task or binstub to start your worker, replace the CMD with the appropriate command, e.g. `CMD ["bundle", "exec", "rake", "hatchet:worker"]`.
```
---
# Autoscaling Workers
Hatchet provides a Task Stats API that enables you to implement autoscaling for your worker pools. By querying real-time queue depths and task distribution, you can dynamically scale workers based on actual workload demand.
## Task Stats API
The Task Stats endpoint returns current statistics for queued and running tasks across your tenant, broken down by task name, queue, and concurrency group.
### Endpoint
```
GET /api/v1/tenants/{tenantId}/task-stats
```
### Authentication
The endpoint requires Bearer token authentication using a valid API token:
```
Authorization: Bearer
```
### Response Format
The response is a JSON object keyed by task name, with each task containing statistics for queued and running states:
```json
{
"my-task": {
"queued": {
"total": 150,
"queues": {
"my-task:default": 100,
"my-task:priority": 50
},
"concurrency": [
{
"expression": "input.user_id",
"type": "GROUP_ROUND_ROBIN",
"keys": {
"user-123": 10,
"user-456": 15
}
}
],
"oldest": "2024-01-15T10:30:00Z"
},
"running": {
"total": 25,
"oldest": "2024-01-15T10:25:00Z",
"concurrency": []
}
}
}
```
Each task stat includes:
- **total**: The total count of tasks in this state
- **concurrency**: Distribution across concurrency groups (if concurrency limits are configured)
- **oldest**: Timestamp of the oldest task in the specified state
These are available only for `queued` tasks:
- **queues**: A breakdown of task counts by queue name
### Example Usage
```bash
curl -H "Authorization: Bearer your-api-token-here" \
https://cloud.onhatchet.run/api/v1/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/task-stats
```
## Autoscaling with KEDA
[KEDA](https://keda.sh) (Kubernetes Event-driven Autoscaling) can use the Task Stats API to automatically scale your worker deployments based on queue depth.
### Setting Up a KEDA ScaledObject
Create a `ScaledObject` that queries the Hatchet Task Stats API and scales your worker deployment based on the number of queued tasks:
```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: hatchet-worker-scaler
spec:
scaleTargetRef:
name: hatchet-worker
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: metrics-api
metadata:
targetValue: "100"
url: "https://cloud.onhatchet.run/api/v1/tenants/YOUR_TENANT_ID/task-stats?taskNames=my-task"
valueLocation: "my-task.queued.total"
authMode: "bearer"
authenticationRef:
name: hatchet-api-token
---
apiVersion: v1
kind: Secret
metadata:
name: hatchet-api-token
type: Opaque
stringData:
token: "your-api-token-here"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: hatchet-api-token
spec:
secretTargetRef:
- parameter: token
name: hatchet-api-token
key: token
```
> **Info:** The `valueLocation` field uses JSONPath-style notation to extract a specific
> value from the response. Adjust `my-task` to match your actual task name.
>
> Pass each task you scale on via the `taskNames` query parameter so the
> endpoint guarantees the JSONPath resolves with `0` when no work is queued.
> Without it, the response can be `{}` and KEDA's metrics-api scaler will fail
> to scale from zero.
### Scaling Based on Multiple Tasks
If you have multiple task types handled by the same worker, you can create multiple triggers or use a custom metrics endpoint that aggregates the totals:
```yaml
triggers:
- type: metrics-api
metadata:
targetValue: "50"
url: "https://cloud.onhatchet.run/api/v1/tenants/YOUR_TENANT_ID/task-stats?taskNames=task-a"
valueLocation: "task-a.queued.total"
authMode: "bearer"
authenticationRef:
name: hatchet-api-token
- type: metrics-api
metadata:
targetValue: "50"
url: "https://cloud.onhatchet.run/api/v1/tenants/YOUR_TENANT_ID/task-stats?taskNames=task-b"
valueLocation: "task-b.queued.total"
authMode: "bearer"
authenticationRef:
name: hatchet-api-token
```
### Scaling Based on Worker Slot Utilization
Instead of scaling on queue depth, you can scale on how utilized your workers' slots are. Hatchet exposes [Prometheus metrics](/self-hosting/prometheus-metrics) that track used and total slots per unique worker label `(key, value)` pair and slot type, so each labeled worker pool can be scaled independently:
- `hatchet_tenant_worker_label_slots{tenant_id, label_key, label_value, slot_type}`: total slots of the slot type across workers with the label pair
- `hatchet_tenant_used_worker_label_slots{tenant_id, label_key, label_value, slot_type}`: used slots of the slot type across workers with the label pair
The `slot_type` label separates a worker's independent slot pools (e.g. `default` and `durable`). Scale on the slot type your tasks consume — usually `default` — so that a large, mostly-idle durable slot pool doesn't mask saturation of the default slots.
Averaging the used and total slot gauges separately over a time window and then dividing gives the average utilization of a worker pool over that window:
```promql
avg_over_time(hatchet_tenant_used_worker_label_slots{tenant_id="", label_key="pool", label_value="gpu", slot_type="default"}[5m])
/
avg_over_time(hatchet_tenant_worker_label_slots{tenant_id="", label_key="pool", label_value="gpu", slot_type="default"}[5m])
```
This query returns a value between 0 and 1. Averaging the numerator and denominator separately (rather than averaging the ratio) keeps the result correct when workers scale up or down within the window.
With [KEDA's Prometheus scaler](https://keda.sh/docs/latest/scalers/prometheus/), you can scale a worker deployment up when average utilization exceeds 75%:
```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: gpu-worker-scaler
spec:
scaleTargetRef:
name: gpu-worker
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
threshold: "0.75"
query: |
avg_over_time(hatchet_tenant_used_worker_label_slots{tenant_id="", label_key="pool", label_value="gpu", slot_type="default"}[5m])
/
avg_over_time(hatchet_tenant_worker_label_slots{tenant_id="", label_key="pool", label_value="gpu", slot_type="default"}[5m])
```
Workers that do not carry the targeted label do not affect the query. To autoscale on a tenant's entire worker fleet regardless of labels, use the per-worker gauges instead:
```promql
sum by (tenant_id) (avg_over_time(hatchet_tenant_used_worker_slots{tenant_id=""}[5m]))
/
sum by (tenant_id) (avg_over_time(hatchet_tenant_worker_slots{tenant_id=""}[5m]))
```
### Scaling Based on Queue Backlog by Metadata Tag
To scale on queue depth via Prometheus instead of the Task Stats API, Hatchet exposes queue size gauges, including one grouped by the additional metadata `(key, value)` pairs on queued tasks. Only metadata keys prefixed with `prom_` are exported. If your GPU workloads tag their workflow runs with additional metadata (e.g. `prom_pool: gpu`), the GPU backlog can drive its own scaler:
```promql
sum(hatchet_tenant_additional_metadata_queue_size{tenant_id="", key="prom_pool", value="gpu"}) or vector(0)
```
The `or vector(0)` matters for scale-to-zero: once a queue drains, its series is removed shortly after reporting zero. There is also `hatchet_tenant_queue_size{tenant_id, queue, workflow_name}` for scaling on a specific queue or workflow's backlog. Both gauges are polled from the database every 15 seconds.
---
# Sticky Worker Assignment (Beta)
> **Info:** This feature is currently in beta and may be subject to change.
Sticky assignment is a task property that allows you to specify that all child tasks should be assigned to the same worker for the duration of its execution. This can be useful in situations like when you need to maintain expensive local memory state across multiple tasks in a workflow or ensure that certain tasks are processed by the same worker for consistency.
> **Warning:** This feature is only compatible with long lived workers, and not webhook
> workers.
## Setting Sticky Assignment
Sticky assignment is set on the task level by adding the `sticky` property to the task definition. When a task is marked as sticky, all steps within that task will be assigned to the same worker for the duration of the task execution.
> **Warning:** While sticky assignment can be useful in certain scenarios, it can also
> introduce potential bottlenecks if the assigned worker becomes unavailable, or
> if local state is not maintained when the job is picked up. Be sure to
> consider the implications of sticky assignment when designing your tasks and
> have a plan in place to handle local state issues.
There are two strategies for setting sticky assignment for [DAG](/v1/directed-acyclic-graphs) workflows:
- `SOFT`: All tasks in the workflow will attempt to be assigned to the same worker, but if that worker is unavailable, it will be assigned to another worker.
- `HARD`: All tasks in the workflow will only be assigned to the same worker. If that worker is unavailable, the workflow run will not be assigned to another worker and will remain in a pending state until the original worker becomes available or timeout is reached. (See [Scheduling Timeouts](/v1/timeouts#task-level-timeouts))
#### Ruby
```python
sticky_workflow = hatchet.workflow(
name="StickyWorkflow",
# 👀 Specify a sticky strategy when declaring the workflow
sticky=StickyStrategy.SOFT,
)
@sticky_workflow.task()
def step1a(input: EmptyModel, ctx: Context) -> dict[str, str | None]:
return {"worker": ctx.worker_id}
@sticky_workflow.task()
def step1b(input: EmptyModel, ctx: Context) -> dict[str, str | None]:
return {"worker": ctx.worker_id}
```
#### Tab 2
```typescript
export const sticky = hatchet.task({
name: 'sticky',
retries: 3,
sticky: StickyStrategy.SOFT,
fn: async (_, ctx) => {
// specify a child workflow to run on the same worker
const result = await child.run(
{
N: 1,
},
{ sticky: true }
);
return {
result,
};
},
});
```
#### Tab 3
```go
func StickyDag(client *hatchet.Client) *hatchet.Workflow {
stickyDag := client.NewWorkflow("sticky-dag",
hatchet.WithWorkflowStickyStrategy(types.StickyStrategy_SOFT),
)
_ = stickyDag.NewTask("sticky-task",
func(ctx worker.HatchetContext, input StickyInput) (interface{}, error) {
workerId := ctx.Worker().ID()
return &StickyResult{
Result: workerId,
}, nil
},
)
_ = stickyDag.NewTask("sticky-task-2",
func(ctx worker.HatchetContext, input StickyInput) (interface{}, error) {
workerId := ctx.Worker().ID()
return &StickyResult{
Result: workerId,
}, nil
},
)
return stickyDag
}
```
#### Tab 4
```ruby
STICKY_WORKFLOW = HATCHET.workflow(
name: "StickyWorkflow",
# Specify a sticky strategy when declaring the workflow
sticky: :soft
)
STEP1A = STICKY_WORKFLOW.task(:step1a) do |input, ctx|
{ "worker" => ctx.worker.id }
end
STEP1B = STICKY_WORKFLOW.task(:step1b) do |input, ctx|
{ "worker" => ctx.worker.id }
end
```
In this example, the `sticky` property is set to `SOFT`, which means that the task will attempt to be assigned to the same worker for the duration of its execution. If the original worker is unavailable, the task will be assigned to another worker.
## Sticky Child Tasks
It is possible to spawn child tasks on the same worker as the parent task by setting the `sticky` property to `true` in the `run` method options. This can be useful when you need to maintain local state across multiple tasks or ensure that child tasks are processed by the same worker for consistency.
However, the child task must:
1. Specify a `sticky` strategy in the child task's definition
2. Be registered with the same worker as the parent task
If either condition is not met, an error will be thrown when the child task is spawned.
#### Ruby
```python
sticky_child_workflow = hatchet.workflow(
name="StickyChildWorkflow", sticky=StickyStrategy.SOFT
)
@sticky_workflow.task(parents=[step1a, step1b])
async def step2(input: EmptyModel, ctx: Context) -> dict[str, str | None]:
ref = await sticky_child_workflow.aio_run(
sticky=True,
wait_for_result=False,
)
await ref.aio_result()
return {"worker": ctx.worker_id}
@sticky_child_workflow.task()
def child(input: EmptyModel, ctx: Context) -> dict[str, str | None]:
return {"worker": ctx.worker_id}
```
#### Tab 2
```typescript
export const sticky = hatchet.task({
name: 'sticky',
retries: 3,
sticky: StickyStrategy.SOFT,
fn: async (_, ctx) => {
// specify a child workflow to run on the same worker
const result = await child.run(
{
N: 1,
},
{ sticky: true }
);
return {
result,
};
},
});
```
#### Tab 3
```go
func Sticky(client *hatchet.Client) *hatchet.StandaloneTask {
sticky := client.NewStandaloneTask("sticky-task",
func(ctx worker.HatchetContext, input StickyInput) (*StickyResult, error) {
// Run a child workflow on the same worker
childWorkflow := Child(client)
childResult, err := childWorkflow.Run(ctx, ChildInput{N: 1}, hatchet.WithRunSticky(true))
if err != nil {
return nil, err
}
var childOutput ChildResult
err = childResult.Into(&childOutput)
if err != nil {
return nil, err
}
return &StickyResult{
Result: fmt.Sprintf("child-result-%s", childOutput.Result),
}, nil
},
)
return sticky
}
```
#### Tab 4
```ruby
STICKY_CHILD_WORKFLOW = HATCHET.workflow(
name: "StickyChildWorkflow",
sticky: :soft
)
STICKY_WORKFLOW.task(:step2, parents: [STEP1A, STEP1B]) do |input, ctx|
ref = STICKY_CHILD_WORKFLOW.run_no_wait(
options: Hatchet::TriggerWorkflowOptions.new(sticky: true)
)
ref.result
{ "worker" => ctx.worker.id }
end
STICKY_CHILD_WORKFLOW.task(:child) do |input, ctx|
{ "worker" => ctx.worker.id }
end
```
---
# Worker Affinity Assignment (Beta)
> **Info:** This feature is currently in beta and may be subject to change.
It is often desirable to assign workflows to specific workers based on certain criteria, such as worker capabilities, resource availability, or location. Worker affinity allows you to specify that a workflow should be assigned to a specific worker based on worker label state. Labels can be set dynamically on workers to reflect their current state, such as a specific model loaded into memory or specific disk requirements.
Specific tasks can then specify desired label state to ensure that workflows are assigned to workers that meet specific criteria. If no worker meets the specified criteria, the task run will remain in a pending state until a suitable worker becomes available or the task is cancelled. (See [Scheduling Timeouts](/v1/timeouts#task-level-timeouts))
## Specifying Worker Labels
Labels can be set on workers when they are registered with Hatchet. Labels are key-value pairs that can be used to specify worker capabilities, resource availability, or other criteria that can be used to match workflows to workers. Values can be strings or numbers, and multiple labels can be set on a worker.
#### Python
```python
worker = hatchet.worker(
"affinity-worker",
slots=10,
labels={
"model": "fancy-ai-model-v2",
"memory": 512,
},
workflows=[affinity_worker_workflow],
)
worker.start()
```
#### Typescript
```typescript
const workflow = hatchet.workflow({
name: 'affinity-workflow',
description: 'test',
});
workflow.task({
name: 'step1',
fn: async (_, ctx) => {
const results = [];
for (let i = 0; i < 50; i++) {
const result = await childWorkflow.run({});
results.push(result);
}
ctx.logger.info('Spawned 50 child workflows');
ctx.logger.info('Results', { results });
return { step1: 'step1 results!' };
},
});
```
#### Go
```go
worker, err := client.NewWorker("affinity-worker",
hatchet.WithWorkflows(affinityWorkflow),
hatchet.WithSlots(10),
hatchet.WithLabels(map[string]any{
"model": "fancy-ai-model-v2",
"memory": 512,
}),
)
```
#### Ruby
```ruby
def main
worker = HATCHET.worker(
"affinity-worker",
slots: 10,
labels: {
"model" => "fancy-ai-model-v2",
"memory" => 512
},
workflows: [AFFINITY_WORKER_WORKFLOW]
)
worker.start
end
```
## Specifying Step Desired Labels
You can specify desired worker label state for specific tasks in a workflow by setting the `desired_worker_labels` property on the task definition. This property is an object where the keys are the label keys and the values are objects with the following properties:
- `value`: The desired value of the label
- `comparator` (default: `EQUAL`): The comparison operator to use when matching the label value.
- `EQUAL`: The label value must be equal to the desired value
- `NOT_EQUAL`: The label value must not be equal to the desired value
- `GREATER_THAN`: The label value must be greater than the desired value
- `GREATER_THAN_OR_EQUAL`: The label value must be greater than or equal to the desired value
- `LESS_THAN`: The label value must be less than the desired value
- `LESS_THAN_OR_EQUAL`: The label value must be less than or equal to the desired value
- `required` (default: `true`): Whether the label is required for the task to run. If `true`, the task will remain in a pending state until a worker with the desired label state becomes available. If `false`, the worker will be prioritized based on the sum of the highest matching weights.
- `weight` (optional, default: `100`): The weight of the label. Higher weights are prioritized over lower weights when selecting a worker for the task. If multiple workers have the same highest weight, the worker with the highest sum of weights will be selected. Ignored if `required` is `true`.
#### Ruby
```python
affinity_worker_workflow = hatchet.workflow(name="AffinityWorkflow")
@affinity_worker_workflow.task(
desired_worker_labels=[
DesiredWorkerLabel(key="model", value="fancy-ai-model-v2", weight=10),
DesiredWorkerLabel(
key="memory",
value=256,
required=True,
comparator=WorkerLabelComparator.LESS_THAN,
),
],
)
```
#### Tab 2
```typescript
const workflow = hatchet.workflow({
name: 'affinity-workflow',
description: 'test',
});
workflow.task({
name: 'step1',
fn: async (_, ctx) => {
const results = [];
for (let i = 0; i < 50; i++) {
const result = await childWorkflow.run({});
results.push(result);
}
ctx.logger.info('Spawned 50 child workflows');
ctx.logger.info('Results', { results });
return { step1: 'step1 results!' };
},
});
```
#### Tab 3
```go
err = w.RegisterWorkflow(
&worker.WorkflowJob{
On: worker.Events("user:create:affinity"),
Name: "affinity",
Description: "affinity",
Steps: []*worker.WorkflowStep{
worker.Fn(func(ctx worker.HatchetContext) (result *taskOneOutput, err error) {
return &taskOneOutput{
Message: ctx.Worker().ID(),
}, nil
}).
SetName("task-one").
SetDesiredLabels(map[string]*types.DesiredWorkerLabel{
"model": {
Value: "fancy-ai-model-v2",
Weight: 10,
},
"memory": {
Value: 512,
Required: true,
Comparator: types.ComparatorPtr(types.WorkerLabelComparator_GREATER_THAN),
},
}),
},
},
)
```
#### Tab 4
```ruby
AFFINITY_WORKER_WORKFLOW = HATCHET.workflow(name: "AffinityWorkflow")
```
> **Warning:** Use extra care when using worker affinity with [sticky assignment `HARD`
> strategy](/v1/advanced-assignment/sticky-assignment). In this case, it is
> recommended to set desired labels on the first task of the workflow to ensure
> that the workflow is assigned to a worker that meets the desired criteria and
> remains on that worker for the duration of the workflow.
### Dynamic Worker Labels
Labels can also be set dynamically on workers using the `upsertLabels` method. This can be useful when worker state changes over time, such as when a new model is loaded into memory or when a worker's resource availability changes.
#### Ruby
```python
async def step(input: EmptyModel, ctx: Context) -> dict[str, str | None]:
if ctx.worker_labels.get("model") != "fancy-ai-model-v2":
ctx.worker.upsert_labels({"model": "unset"})
# DO WORK TO EVICT OLD MODEL / LOAD NEW MODEL
ctx.worker.upsert_labels({"model": "fancy-ai-model-v2"})
return {"worker": ctx.worker_id}
```
#### Tab 2
```typescript
const childWorkflow = hatchet.workflow({
name: 'child-affinity-workflow',
description: 'test',
});
childWorkflow.task({
name: 'child-step1',
desiredWorkerLabels: {
model: {
value: 'xyz',
required: true,
},
},
fn: async (ctx) => {
return { childStep1: 'childStep1 results!' };
},
});
```
#### Tab 3
```go
err = w.RegisterWorkflow(
&worker.WorkflowJob{
On: worker.Events("user:create:affinity"),
Name: "affinity",
Description: "affinity",
Steps: []*worker.WorkflowStep{
worker.Fn(func(ctx worker.HatchetContext) (result *taskOneOutput, err error) {
model := ctx.Worker().GetLabels()["model"]
if model != "fancy-vision-model" {
ctx.Worker().UpsertLabels(map[string]interface{}{
"model": nil,
})
// Do something to load the model
evictModel();
loadNewModel("fancy-vision-model");
ctx.Worker().UpsertLabels(map[string]interface{}{
"model": "fancy-vision-model",
})
}
return &taskOneOutput{
Message: ctx.Worker().ID(),
}, nil
}).
SetName("task-one").
SetDesiredLabels(map[string]*types.DesiredWorkerLabel{
"model": {
Value: "fancy-vision-model",
Weight: 10,
},
"memory": {
Value: 512,
Required: true,
Comparator: types.WorkerLabelComparator_GREATER_THAN,
},
}),
},
},
)
```
#### Tab 4
```ruby
AFFINITY_WORKER_WORKFLOW.task(
:step,
desired_worker_labels: {
"model" => Hatchet::DesiredWorkerLabel.new(value: "fancy-ai-model-v2", weight: 10),
"memory" => Hatchet::DesiredWorkerLabel.new(
value: 256,
required: true,
comparator: :less_than
)
}
) do |input, ctx|
if ctx.worker.labels["model"] != "fancy-ai-model-v2"
ctx.worker.upsert_labels("model" => "unset")
# DO WORK TO EVICT OLD MODEL / LOAD NEW MODEL
ctx.worker.upsert_labels("model" => "fancy-ai-model-v2")
end
{ "worker" => ctx.worker.id }
end
```
---
# Task Slot Cost
Every worker has a fixed number of slots that limit how many tasks it runs at once, set with the `slots` option on the worker and defaulting to 100. By default a task consumes one slot while it runs. Slot cost lets a task consume more than one slot, so a task that needs more memory or CPU takes up more of a worker's capacity and the worker runs fewer of them at the same time.
## Setting slot cost
Set the slot cost when you define a task. A cost of 5 means the task reserves five slots while it runs.
#### Python
```python
@hatchet.task(slot_cost=5)
def omega(input: EmptyModel, ctx: Context) -> None:
print("heavy work")
@hatchet.task(slot_cost=1)
def weenie(input: EmptyModel, ctx: Context) -> None:
print("light work")
```
#### Typescript
```typescript
import { hatchet } from '../hatchet-client';
export const omega = hatchet.task({
name: 'omega',
slotCost: 5,
fn: async () => {
console.log('heavy work');
},
});
export const weenie = hatchet.task({
name: 'weenie',
slotCost: 1,
fn: async () => {
console.log('light work');
},
});
```
#### Go
```go
omega := client.NewStandaloneTask("omega", func(ctx hatchet.Context, input any) (any, error) {
log.Println("heavy work")
return nil, nil
}, hatchet.WithSlotCost(5))
weenie := client.NewStandaloneTask("weenie", func(ctx hatchet.Context, input any) (any, error) {
log.Println("light work")
return nil, nil
}, hatchet.WithSlotCost(1))
```
On a worker with 100 slots, the omega task at cost 5 and the weenie task at cost 1 draw from the same 100 slots. The worker runs at most 20 omega tasks, or 100 weenie tasks, or any mix whose costs sum to 100.
## How the reservation works
Slot capacity is local to a single worker. A task's slot cost is charged against the one worker that runs it, and a reservation cannot span two workers. A task with cost 5 needs a single worker with 5 free slots. It cannot take 3 slots from one worker and 2 from another, even when the worker pool has capacity in total.
Set the slot count on each worker to at least the largest slot cost you use. If a task's cost is greater than every worker's slot count, the task can never be scheduled. It waits in the queue until its [schedule timeout](/v1/timeouts) and is then cancelled.
> **Info:** Changing a task's slot cost changes its workflow version, so the next
> registration creates a new version. Leaving the cost unset, or setting it to
> 1, keeps the task at one slot and does not change the version.
## Slot cost and concurrency limits
Slot cost and [concurrency limits](/v1/concurrency) solve different problems. A concurrency limit caps how many runs of a key execute at once, and with a static key the limit applies across the whole worker pool. Slot cost does not limit the number of runs. It changes how much of a worker one run consumes.
If you want at most five heavy runs at once, use a concurrency limit. If you want each heavy run to take five times the worker capacity of a light run, use slot cost. The two can be combined on the same task.
---
# Manual Slot Release
The Hatchet execution model sets a number of available slots for running tasks in a workflow. When a task is running, it occupies a slot, and if a worker has no available slots, it will not be able to run any more tasks concurrently.
In some cases, you may have a task in your workflow that is resource-intensive and requires exclusive access to a shared resource, such as a database connection or a GPU compute instance. To ensure that other tasks in the workflow can run concurrently, you can manually release the slot after the resource-intensive task has completed, but the task still has non-resource-intensive work to do (i.e. upload or cleanup).
> **Warning:** This is an advanced feature and should be used with caution. Manually
> releasing the slot can have unintended side effects on system performance and
> concurrency. For example, if the worker running the task dies, the task will
> not be reassigned and will remain in a running state until manually
> terminated.
## Using Manual Slot Release
You can manually release a slot in from within a running task in your workflow using the Hatchet context method `release_slot`:
#### Go
```python
slot_release_workflow = hatchet.workflow(name="SlotReleaseWorkflow")
@slot_release_workflow.task()
def step1(input: EmptyModel, ctx: Context) -> dict[str, str]:
print("RESOURCE INTENSIVE PROCESS")
time.sleep(10)
# 👀 Release the slot after the resource-intensive process, so that other steps can run
ctx.release_slot()
print("NON RESOURCE INTENSIVE PROCESS")
return {"status": "success"}
```
#### Ruby
```go
_ = workflow.NewTask("step1", func(ctx hatchet.Context, _ any) (*StepOutput, error) {
fmt.Println("RESOURCE INTENSIVE PROCESS")
time.Sleep(10 * time.Second)
// Release the slot after the resource-intensive process,
// so that other steps can run on this worker.
if releaseErr := ctx.ReleaseSlot(); releaseErr != nil {
return nil, fmt.Errorf("failed to release slot: %w", releaseErr)
}
fmt.Println("NON RESOURCE INTENSIVE PROCESS")
return &StepOutput{Status: "success"}, nil
})
```
#### Tab 3
```ruby
SLOT_RELEASE_WORKFLOW = HATCHET.workflow(name: "SlotReleaseWorkflow")
SLOT_RELEASE_WORKFLOW.task(:step1) do |input, ctx|
puts "RESOURCE INTENSIVE PROCESS"
sleep 10
# Release the slot after the resource-intensive process, so that other steps can run
ctx.release_slot
puts "NON RESOURCE INTENSIVE PROCESS"
{ "status" => "success" }
end
```
In the above examples, the `release_slot()` method is called after the resource-intensive process has completed. This allows other tasks in the workflow to start executing while the current task continues with non-resource-intensive tasks.
> **Info:** Manually releasing the slot does not terminate the current task. The task will
> continue executing until it completes or encounters an error.
## Use Cases
Some common use cases for Manual Slot Release include:
- Performing data processing or analysis that requires significant CPU, GPU, or memory resources
- Acquiring locks or semaphores to access shared resources
- Executing long-running tasks that don't need to block other tasks after some initial work is done
By utilizing Manual Slot Release, you can optimize the concurrency and resource utilization of your workflows, allowing multiple tasks to run in parallel when possible.
---
# Logging
Hatchet comes with a built-in logging view where you can push logs from your workflows. This is useful for debugging and monitoring your workflows.
#### Ruby
You can use either Python's built-in `logging` package, or the `context.log` method for more control over the logs that are sent.
## Using the built-in `logging` package
You can pass a custom logger to the `Hatchet` class when initializing it. For example:
```python
import logging
from hatchet_sdk import ClientConfig, Hatchet
logging.basicConfig(level=logging.INFO)
root_logger = logging.getLogger()
hatchet = Hatchet(
config=ClientConfig(
logger=root_logger,
),
)
```
It's recommended that you pass the root logger to the `Hatchet` class, as this will ensure that all logs are captured by the Hatchet logger. If you have workflows defined in multiple files, they should be children of the root logger. For example, with the following file structure:
```
workflows/
workflow.py
client.py
worker.py
workflow.py
```
You should pass the root logger to the `Hatchet` class in `client.py`:
```python
import logging
from hatchet_sdk import ClientConfig, Hatchet
logging.basicConfig(level=logging.INFO)
root_logger = logging.getLogger()
hatchet = Hatchet(
config=ClientConfig(
logger=root_logger,
),
)
```
And then in `workflows/workflow.py`, you should create a child logger:
```python
import logging
import time
from examples.logger.client import hatchet
from hatchet_sdk import Context, EmptyModel
logger = logging.getLogger(__name__)
logging_workflow = hatchet.workflow(
name="LoggingWorkflow",
)
@logging_workflow.task()
def root_logger(input: EmptyModel, ctx: Context) -> dict[str, str]:
for i in range(12):
logger.info(f"executed step1 - {i}")
logger.info({"step1": "step1"})
time.sleep(0.1)
return {"status": "success"}
```
## Using the `context.log` method
You can also use the `context.log` method to log messages from your workflows. This method is available on the `Context` object that is passed to each task in your workflow. For example:
```python
@logging_workflow.task()
def context_logger(input: EmptyModel, ctx: Context) -> dict[str, str]:
for i in range(12):
ctx.log(f"executed step1 - {i}")
ctx.log({"step1": "step1"})
time.sleep(0.1)
return {"status": "success"}
```
Each task is currently limited to 1000 log lines.
#### Tab 2
In TypeScript, there are two options for logging from your tasks. The first is to use the `ctx.log()` method (from the `Context`) to send logs:
```typescript
const workflow = hatchet.workflow({
name: 'logger-example',
description: 'test',
on: {
event: 'user:create',
},
});
workflow.task({
name: 'logger-step1',
fn: async (_, ctx) => {
// log in a for loop
for (let i = 0; i < 10; i++) {
ctx.logger.info(`log message ${i}`);
await sleep(200);
}
return { step1: 'completed step run' };
},
});
```
This has the benefit of being easy to use out of the box (no setup required!), but it's limited in its flexibiliy and how pluggable it is with your existing logging setup.
Hatchet also allows you to "bring your own" logger when you define a workflow:
```typescript
const logger = pino();
class PinoLogger implements Logger {
logLevel: LogLevel;
context: string;
constructor(context: string, logLevel: LogLevel = 'DEBUG') {
this.logLevel = logLevel;
this.context = context;
}
debug(message: string, extra?: JsonObject): void {
logger.debug(extra, message);
}
info(message: string, extra?: JsonObject): void {
logger.info(extra, message);
}
green(message: string, extra?: JsonObject): void {
logger.info(extra, `%c${message}`);
}
warn(message: string, error?: Error, extra?: JsonObject): void {
logger.warn(extra, `${message} ${error}`);
}
error(message: string, error?: Error, extra?: JsonObject): void {
logger.error(extra, `${message} ${error}`);
}
// optional util method
util(key: string, message: string, extra?: JsonObject): void {
// for example you may want to expose a trace method
if (key === 'trace') {
logger.info(extra, 'trace');
}
}
}
const hatchet = Hatchet.init({
log_level: 'DEBUG',
logger: (ctx, level) => new PinoLogger(ctx, level),
});
```
In this example, we create Pino logger that implement's Hatchet's `Logger` interface and pass it to the Hatchet client constructor. We can then use that logger in our steps:
```typescript
const workflow = hatchet.workflow({
name: 'logger-example',
description: 'test',
on: {
event: 'user:create',
},
});
workflow.task({
name: 'logger-step1',
fn: async (_, ctx) => {
// log in a for loop
for (let i = 0; i < 10; i++) {
ctx.logger.info(`log message ${i}`);
await sleep(200);
}
return { step1: 'completed step run' };
},
});
```
#### Tab 3
```ruby
require "hatchet-sdk"
require "logger"
logger = Logger.new($stdout)
logger.level = Logger::INFO
HATCHET = Hatchet::Client.new(debug: true) unless defined?(HATCHET)
LOGGING_WORKFLOW = HATCHET.workflow(name: "LoggingWorkflow")
LOGGING_WORKFLOW.task(:root_logger) do |input, ctx|
12.times do |i|
logger.info("executed step1 - #{i}")
logger.info({ "step1" => "step1" }.inspect)
sleep 0.1
end
{ "status" => "success" }
end
```
```ruby
LOGGING_WORKFLOW.task(:context_logger) do |input, ctx|
12.times do |i|
ctx.log("executed step1 - #{i}")
ctx.log({ "step1" => "step1" }.inspect)
sleep 0.1
end
{ "status" => "success" }
end
```
---
# OpenTelemetry
Hatchet supports exporting traces from your tasks to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to improve visibility into your Hatchet tasks.
> **Info:** This page is about tracing your application, task, and workflow code. To
> export traces emitted by Hatchet's own services on a self-hosted instance, see
> [Internal OpenTelemetry traces](/self-hosting/opentelemetry).
## Setup
#### Python
Install the `otel` extra:
```bash
pip install hatchet-sdk[otel]
```
Then create the instrumentor and call `instrument()`:
```python
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor
HatchetInstrumentor().instrument()
```
By default, `HatchetInstrumentor` creates a `TracerProvider` that sends spans to the Hatchet engine's OTLP collector. You can also pass your own `TracerProvider`:
```python
HatchetInstrumentor(
tracer_provider=your_tracer_provider,
).instrument()
```
### Options
Option, Type, Default, Description
`tracer_provider`, `TracerProvider`, —, Custom TracerProvider. If not set, one is created automatically.
`enable_hatchet_otel_collector`, `bool`, `True`, Send traces to the Hatchet engine's OTLP collector.
`schedule_delay_millis`, `int`, —, Delay between consecutive exports of the BatchSpanProcessor.
`max_export_batch_size`, `int`, —, Maximum batch size for the BatchSpanProcessor.
`max_queue_size`, `int`, —, Maximum queue size for the BatchSpanProcessor.
#### TypeScript
Install the required OpenTelemetry packages:
```bash
npm install @opentelemetry/api @opentelemetry/instrumentation @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-grpc
```
Register the instrumentor before creating any Hatchet clients or workers:
```typescript
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
import { HatchetInstrumentor } from "@hatchet-dev/typescript-sdk/opentelemetry";
registerInstrumentations({
instrumentations: [new HatchetInstrumentor()],
});
```
By default, `HatchetInstrumentor` sends spans to the Hatchet engine's OTLP collector. You can disable this with `enableHatchetCollector: false`.
### Options
Option, Type, Default, Description
`enableHatchetCollector`, `boolean`, `true`, Send traces to the Hatchet engine's OTLP collector.
`clientConfig`, `object`, —, Override Hatchet client config for the collector connection.
`includeTaskNameInSpanName`, `boolean`, `false`, Append the task action ID to the `hatchet.start_step_run` span name.
`excludedAttributes`, `array`, `[]`, List of `hatchet.*` attribute keys to exclude from spans.
#### Go
Import the `opentelemetry` package (commonly aliased as `hatchetotel`):
```go
import hatchetotel "github.com/hatchet-dev/hatchet/sdks/go/opentelemetry"
```
Create the instrumentor, then register its middleware on the worker:
```go
instrumentor, err := hatchetotel.NewInstrumentor()
if err != nil {
log.Fatalf("failed to create instrumentor: %v", err)
}
worker.Use(instrumentor.Middleware())
```
By default, `NewInstrumentor` creates a `TracerProvider` that sends spans to the Hatchet engine's OTLP collector.
Remember to shut down the instrumentor on exit to flush remaining spans:
```go
defer instrumentor.Shutdown(context.Background())
```
### Options
Option, Description
`hatchetotel.WithTracerProvider(tp)`, Use a custom `*sdktrace.TracerProvider` instead of creating a new one.
`hatchetotel.DisableHatchetCollector()`, Disable sending traces to the Hatchet engine's OTLP collector.
`hatchetotel.WithBatchSpanProcessorOptions(...)`, Configure the `BatchSpanProcessor` for the Hatchet collector.
#### Ruby
> **Info:** OpenTelemetry support for the Ruby SDK is coming soon.
## Spans
By default, Hatchet creates spans at the following points in the lifecycle of a task run:
1. **Producer spans** — when a trigger is called on the client side (e.g. `run()`, `runNoWait()`, `push()`, `schedule()`).
2. **Consumer spans** — when a worker handles a task run, such as starting to run the task (`hatchet.start_step_run`) or cancelling a task (`hatchet.cancel_step_run`).
### Span Names
Span Name, Kind, Description
`hatchet.start_step_run`, `CONSUMER`, Worker begins executing a task.
`hatchet.cancel_step_run`, `CONSUMER`, Worker cancels a running task.
`hatchet.run_workflow`, `PRODUCER`, Client triggers a workflow run.
`hatchet.run_workflows`, `PRODUCER`, Client triggers multiple workflow runs.
`hatchet.schedule_workflow`, `PRODUCER`, Client creates a scheduled workflow run.
`hatchet.push_event`, `PRODUCER`, Client pushes an event.
`hatchet.bulk_push_event`, `PRODUCER`, Client pushes events in bulk.
`hatchet.durable.wait_for`, `INTERNAL`, Durable task waits for a signal/condition.
### Span Attributes
All spans include an `instrumentor` attribute set to `"hatchet"`. Consumer spans (`hatchet.start_step_run`) include the following `hatchet.*` attributes:
Attribute, Type, Description
`hatchet.tenant_id`, `string`, The tenant ID for the task run.
`hatchet.worker_id`, `string`, The worker handling the task.
`hatchet.workflow_run_id`, `string`, The workflow run ID.
`hatchet.step_run_id`, `string`, The task run ID.
`hatchet.step_id`, `string`, The task ID.
`hatchet.workflow_name`, `string`, The workflow name.
`hatchet.action_name`, `string`, The action ID (format: `workflowName:taskName`).
`hatchet.step_name`, `string`, The human-readable task name.
`hatchet.retry_count`, `int`, Current retry attempt (0-indexed).
`hatchet.workflow_id`, `string`, The workflow definition ID (if available).
`hatchet.workflow_version_id`, `string`, The workflow version ID (if available).
`hatchet.parent_workflow_run_id`, `string`, Parent workflow run ID (for child workflows).
`hatchet.child_workflow_index`, `int`, Child workflow index (for child workflows).
`hatchet.child_workflow_key`, `string`, Child workflow key (for child workflows).
Producer spans (`hatchet.run_workflow`) include `hatchet.step_name` (the workflow being triggered) and, on success, `hatchet.child_workflow_run_id`.
### Context Propagation
All SDKs:
1. Automatically inject W3C `traceparent` into `additionalMetadata` on producer spans, so consumer spans on the worker become children of the trigger span.
2. Provide an `HatchetAttributeSpanProcessor` that propagates `hatchet.*` attributes from the parent task run span to all child spans created within the task. This means any custom spans you create inside a task function will automatically carry the same `hatchet.*` attributes.
---
# Worker Health Checks
The Python SDK allows you to enable and ping a healthcheck to check on the status of your worker.
### Usage
First, set the `HATCHET_CLIENT_WORKER_HEALTHCHECK_ENABLED` environment variable to `True`. Once that flag is set, two health check endpoints will be available (on port `8001` by default):
1. `/health` - Returns **200** when the worker listener is healthy, otherwise **503** with body `{"status":"HEALTHY"}` or `{"status":"UNHEALTHY"}`.
2. `/metrics` - A metrics endpoint intended to be used by a monitoring system like Prometheus.
### Custom Port
You can set a custom port with the `HATCHET_CLIENT_WORKER_HEALTHCHECK_PORT` environment variable, e.g. `HATCHET_CLIENT_WORKER_HEALTHCHECK_PORT=8002`.
### Event loop blocked threshold
If the worker listener process event loop becomes blocked for longer than a threshold, `/health` will return **503**.
You can configure this threshold (in seconds) with:
- `HATCHET_CLIENT_WORKER_HEALTHCHECK_EVENT_LOOP_BLOCK_THRESHOLD_SECONDS` (default: `5.0`)
#### Example request to `/health`:
```bash
curl localhost:8001/health
{"status":"HEALTHY"}
```
#### Example request to `/metrics`:
```bash
curl localhost:8001/metrics
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 18782.0
python_gc_objects_collected_total{generation="1"} 4907.0
python_gc_objects_collected_total{generation="2"} 244.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 308.0
python_gc_collections_total{generation="1"} 27.0
python_gc_collections_total{generation="2"} 2.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="10",patchlevel="15",version="3.10.15"} 1.0
# HELP hatchet_worker_listener_health_my_worker Listener health (1 healthy, 0 unhealthy)
# TYPE hatchet_worker_listener_health_my_worker gauge
hatchet_worker_listener_health_my_worker 1.0
# HELP hatchet_worker_event_loop_lag_seconds_my_worker Event loop lag in seconds (listener process)
# TYPE hatchet_worker_event_loop_lag_seconds_my_worker gauge
hatchet_worker_event_loop_lag_seconds_my_worker 0.0
```
#### Example Prometheus Configuration for `/metrics`:
```yaml
scrape_configs:
- job_name: "hatchet"
scrape_interval: 5s
static_configs:
- targets: ["localhost:8001"]
```
#### Example Prometheus Query
An example query to check if the worker is healthy might look something like:
```
(hatchet_worker_listener_health_my_worker{instance="localhost:8001", job="hatchet"}) or vector(0)
```
---
# Prometheus Metrics
> **Info:** Only available in the Enterprise tier on Hatchet Cloud, [reach
> out](https://cal.com/team/hatchet/talk-to-us) to upgrade.
Hatchet exports Prometheus Metrics for your tenant which can be scraped with services like Grafana and DataDog.
> **Info:** For a full list of available metrics, setup instructions, and example PromQL
> queries, see the [Prometheus Metrics self-hosting
> guide](/self-hosting/prometheus-metrics).
## Tenant Metrics
Metrics for individual tenants are available in Prometheus Text Format via a REST API endpoint.
### Endpoint
```
GET /api/v1/tenants/{tenantId}/prometheus-metrics
```
### Authentication
The endpoint requires Bearer token authentication using a valid API token:
```
Authorization: Bearer
```
### Response Format
The response is returned in standard Prometheus Text Format, including:
- HELP comments describing each metric
- TYPE declarations (counter, gauge, etc.)
- Metric samples with labels and values
### Example Usage
```bash
curl -H "Authorization: Bearer your-api-token-here" \
https://cloud.onhatchet.run/api/v1/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/prometheus-metrics
```
---
# Additional Metadata
Hatchet allows you to attach arbitrary key-value string pairs to events and task runs, which can be used for filtering, searching, or any other lookup purposes. This additional metadata is not part of the event payload or task input data but provides supplementary information for better organization and discoverability.
> **Info:** Additional metadata can be added to `Runs`, `Scheduled Runs`, `Cron Runs`, and
> `Events`. The data is propagated from parents to children or from events to
> runs.
You can attach additional metadata when pushing events or triggering task runs using the Hatchet client libraries:
#### Event Push
#### Ruby
```python
hatchet.event.push(
"user:create",
{"userId": "1234", "should_skip": False},
additional_metadata={"source": "api"}, # Arbitrary key-value pair
)
```
#### Tab 2
```typescript
const withMetadata = await hatchet.events.push(
'user:create',
{
test: 'test',
},
{
additionalMetadata: {
source: 'api', // Arbitrary key-value pair
},
}
);
```
#### Tab 3
```go
err = client.Events().Push(
context.Background(),
"user:create",
Input{Message: "hello"},
v0Client.WithEventMetadata(
map[string]string{"version": "1.0.0"},
),
)
if err != nil {
log.Fatalf("failed to push event: %v", err)
}
```
#### Tab 4
```ruby
HATCHET.event.push(
"user:create",
{ "userId" => "1234", "should_skip" => false },
additional_metadata: { "source" => "api" }
)
```
#### Task Run Trigger
#### Ruby
```python
simple.run(
additional_metadata={"source": "api"}, # Arbitrary key-value pair
)
```
#### Tab 2
```typescript
const withMetadata = simple.run(
{
Message: 'HeLlO WoRlD',
},
{
additionalMetadata: {
source: 'api', // Arbitrary key-value pair
},
}
);
```
#### Tab 3
```go
_, err = client.Run(
context.Background(),
"my-workflow",
Input{Message: "hello"},
hatchet.WithRunMetadata(
map[string]string{"version": "1.0.0"},
),
)
if err != nil {
log.Fatalf("failed to run workflow: %v", err)
}
```
#### Tab 4
```ruby
SIMPLE.run(
{},
options: Hatchet::TriggerWorkflowOptions.new(
additional_metadata: { "source" => "api" }
)
)
```
## Filtering in the Dashboard
Once you've attached additional metadata to events or task runs, this data will be available in the Event and Task Run list views in the Hatchet dashboard. You can use the filter input field to search for events or task runs based on the additional metadata key-value pairs you've attached.
For example, you can filter events by the `source` metadata keys to quickly find events originating from a specific source or environment.
## Filtering Semantics
When filtering **runs** with `additional_metadata`, Hatchet returns runs that match **any** of the provided pairs (logical `OR`), not all of them.
> **Info:** For example, filtering runs with `{ first_name: "john", last_name: "doe" }`
> returns runs whose metadata matches *either* `first_name: "john"` *or*
> `last_name: "doe"`, not only runs that match both.
Event filtering behaves differently: filtering **events** with multiple `additional_metadata` pairs requires all pairs to match (logical `AND`), since events use `JSONB` containment under the hood.
> **Info:** For example, filtering events with `{ first_name: "john", last_name: "doe" }`
> returns only events whose metadata matches *both* `first_name: "john"` *and*
> `last_name: "doe"`.
## Use Cases
Some common use cases for additional metadata include:
- Tagging events or task runs with environment information (e.g., `production`, `staging`, `development`)
- Specifying the source or origin of events (e.g., `api`, `webhook`, `manual`)
- Categorizing events or task runs based on business-specific criteria (e.g., `priority`, `region`, `product`)
By leveraging additional metadata, you can enhance the organization, searchability, and discoverability of your events and task runs within Hatchet.
---
# Middleware & Dependency Injection
Middleware lets you run logic **before** and **after** every task on a client, without touching individual task definitions. Common uses include injecting request IDs, enriching inputs with shared data, encrypting/decrypting payloads, and normalizing or augmenting outputs.
This feature is experimental, and middleware hook signatures may change in
future releases.
#### Python
Hatchet's Python SDK uses FastAPI-style dependency injection to run logic
before tasks and inject the results as parameters. Dependencies are declared
as functions and wired into tasks with `Depends`.
#### Typescript
Middleware hooks are registered on the client with `withMiddleware` and are
fully type-safe — TypeScript sees the union of fields from the task input
type and any values returned by `before` hooks, and similarly for task
outputs and `after` hooks.
#### Go
> **Info:** Middleware support for the Go SDK is coming soon. Join our
> [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
In Ruby, this pattern uses callable objects (lambdas/procs) passed as `deps`
when defining tasks. Dependencies are evaluated before each task run and
made available via `ctx.deps`.
## Defining Middleware
#### Python
Define your dependency functions — they receive the workflow input and context, and their return values are injected into the task as parameters.
```python
async def async_dep(input: EmptyModel, ctx: Context) -> str:
return ASYNC_DEPENDENCY_VALUE
def sync_dep(input: EmptyModel, ctx: Context) -> str:
return SYNC_DEPENDENCY_VALUE
@asynccontextmanager
async def async_cm_dep(
input: EmptyModel, ctx: Context, async_dep: Annotated[str, Depends(async_dep)]
) -> AsyncGenerator[str, None]:
try:
yield ASYNC_CM_DEPENDENCY_VALUE + "_" + async_dep
finally:
pass
@contextmanager
def sync_cm_dep(
input: EmptyModel, ctx: Context, sync_dep: Annotated[str, Depends(sync_dep)]
) -> Generator[str, None, None]:
try:
yield SYNC_CM_DEPENDENCY_VALUE + "_" + sync_dep
finally:
pass
@contextmanager
def base_cm_dep(input: EmptyModel, ctx: Context) -> Generator[str, None, None]:
try:
yield CHAINED_CM_VALUE
finally:
pass
def chained_dep(
input: EmptyModel, ctx: Context, base_cm: Annotated[str, Depends(base_cm_dep)]
) -> str:
return "chained_" + base_cm
@asynccontextmanager
async def base_async_cm_dep(
input: EmptyModel, ctx: Context
) -> AsyncGenerator[str, None]:
try:
yield CHAINED_ASYNC_CM_VALUE
finally:
pass
async def chained_async_dep(
input: EmptyModel,
ctx: Context,
base_async_cm: Annotated[str, Depends(base_async_cm_dep)],
) -> str:
return "chained_" + base_async_cm
```
#### Typescript
Create a client and attach middleware with `before` and `after` hooks.
- **`before(input, ctx)`** runs before the task. Its return value **replaces** the task input.
- **`after(output, ctx, input)`** runs after the task. Its return value **replaces** the task output.
```typescript
import { HatchetClient, HatchetMiddleware } from '@hatchet/v1';
export type GlobalInputType = {
first: number;
second: number;
};
export type GlobalOutputType = {
extra: number;
};
const myMiddleware = {
before: (input, ctx) => {
ctx.logger.info('before', { first: input.first });
return { ...input, dependency: 'abc-123' };
},
after: (output, ctx, input) => {
return { ...output, additionalData: 2 };
},
} satisfies HatchetMiddleware;
export const hatchetWithMiddleware = HatchetClient.init<
GlobalInputType,
GlobalOutputType
>().withMiddleware(myMiddleware);
```
**Spread the original value if you want to keep it.** The return value of each hook **replaces** the input (or output) entirely — it does not shallow-merge. If you omit `...input` in a `before` hook, the original fields are lost. The same applies to `...output` in an `after` hook.
```typescript
// ✅ Keeps original fields and adds `requestId`
before: (input) => ({ ...input, requestId: crypto.randomUUID() })
// ❌ Replaces input entirely — task only receives { requestId }
before: (input) => ({ requestId: crypto.randomUUID() })
```
### Chaining Middleware
You can chain multiple `.withMiddleware()` calls to run hooks in sequence. Each `before` hook receives the return value of the previous `before` hook (or the original input for the first hook), and each `after` hook receives the return value of the previous `after` hook.
```typescript
const firstMiddleware = {
before: (input, ctx) => {
ctx.logger.info('before', { first: input.first });
return { ...input, dependency: 'abc-123' };
},
after: (output, ctx, input) => {
return { ...output, firstExtra: 3 };
},
} satisfies HatchetMiddleware;
const secondMiddleware = {
before: (input, ctx) => {
ctx.logger.info('before', { dependency: input.dependency }); // available from previous middleware
return { ...input, anotherDep: true };
},
after: (output, ctx, input) => {
return { ...output, secondExtra: 4 };
},
} satisfies HatchetMiddleware;
export const hatchetWithMiddlewareChaining = HatchetClient.init()
.withMiddleware(firstMiddleware)
.withMiddleware(secondMiddleware);
```
#### Go
> **Info:** Middleware support for the Go SDK is coming soon. Join our [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
Define your dependencies as callable objects (lambdas). They receive the input, context, and optionally a hash of previously resolved dependencies for chaining.
```ruby
sync_dep = ->(_input, _ctx) { SYNC_DEPENDENCY_VALUE }
async_dep = ->(_input, _ctx) { ASYNC_DEPENDENCY_VALUE }
sync_cm_dep = lambda { |_input, _ctx, deps|
"#{SYNC_CM_DEPENDENCY_VALUE}_#{deps[:sync_dep]}"
}
async_cm_dep = lambda { |_input, _ctx, deps|
"#{ASYNC_CM_DEPENDENCY_VALUE}_#{deps[:async_dep]}"
}
chained_dep = ->(_input, _ctx, deps) { "chained_#{CHAINED_CM_VALUE}" }
chained_async_dep = ->(_input, _ctx, deps) { "chained_#{CHAINED_ASYNC_CM_VALUE}" }
```
## How Middleware Executes
#### Python
Dependencies are resolved before each task execution. Each dependency function receives the original workflow input and the task context, and its return value is injected as a named parameter to the task function.
#### Typescript
When a task runs, the worker applies middleware hooks in this order:
### Before hooks run in registration order
Each `before` hook receives the current input and the task `Context`. Its return value **replaces** the input for the next hook (or the task itself). Returning `undefined` (or `void`) skips replacement and passes the input through unchanged.
### The task function executes
The task receives the final input after all `before` hooks have run.
### After hooks run in registration order
Each `after` hook receives the current output, the task `Context`, and the final input. Its return value **replaces** the output for the next hook (or the final result). Returning `undefined` skips replacement.
Both `before` and `after` hooks can be **async** — return a `Promise` and it will be awaited before proceeding.
> **Info:** If a middleware hook throws an error, the task run fails with that error. There is no built-in error recovery within middleware — use try/catch inside your hooks if you need graceful fallback.
### The `ctx` Parameter
The second parameter of both `before` and `after` hooks is the task `Context` object. This gives middleware access to:
- `ctx.workflowRunId` — the ID of the current workflow run
- `ctx.stepRunId` — the ID of the current step run
- `ctx.log()` — emit structured logs visible in the Hatchet dashboard
- `ctx.cancel()` — cancel the current run from within middleware
### Global Types vs Middleware Types
There are two ways extra fields end up on a task's input:
Mechanism, Set via, Required at call site?, Available at runtime?
**Global input type**, `HatchetClient.init()`, Yes — callers must provide these fields, Yes
**Middleware before hook**, `.withMiddleware({ before })`, No — injected automatically by the worker, Yes
Global input types (`T` in `init()`) represent fields that **callers must supply** when triggering a task. This is useful when you know every task must always receive certain parameters — for example, a `userId` for authentication or a `tenantId` for multi-tenant routing. By declaring these as the global type, TypeScript enforces that every caller provides them.
Middleware `before` hooks, on the other hand, inject fields that are **computed at runtime** (e.g. request IDs, decrypted secrets, fetched config) and are **not** required from callers.
```typescript
type RequiredContext = { userId: string; orgId: string };
const client = HatchetClient.init()
.withMiddleware({
before: (input) => ({
...input,
resolvedAt: Date.now(), // injected, not required from caller
permissions: lookupPerms(input.userId), // derived from global type
}),
});
// Callers MUST provide userId and orgId — TypeScript enforces this
await myTask.run({ userId: 'usr_123', orgId: 'org_456', /* ...task fields */ });
```
#### Go
> **Info:** Middleware support for the Go SDK is coming soon. Join our [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
Dependencies are resolved in the order they are declared in the `deps` hash. Each dependency function can optionally receive already-resolved dependencies as its third argument, enabling chaining.
## Using Middleware in Tasks
#### Python
Inject dependencies into your tasks using `Depends` and type annotations. The dependency results are passed directly as function parameters.
```python
@hatchet.task()
async def async_task_with_dependencies(
_i: EmptyModel,
ctx: Context,
async_dep: Annotated[str, Depends(async_dep)],
sync_dep: Annotated[str, Depends(sync_dep)],
async_cm_dep: Annotated[str, Depends(async_cm_dep)],
sync_cm_dep: Annotated[str, Depends(sync_cm_dep)],
chained_dep: Annotated[str, Depends(chained_dep)],
chained_async_dep: Annotated[str, Depends(chained_async_dep)],
) -> Output:
return Output(
sync_dep=sync_dep,
async_dep=async_dep,
async_cm_dep=async_cm_dep,
sync_cm_dep=sync_cm_dep,
chained_dep=chained_dep,
chained_async_dep=chained_async_dep,
)
```
Your dependency functions must take two positional arguments: the workflow input and the `Context` (the same as any other task).
#### Typescript
Tasks created from a middleware-enabled client automatically receive the merged input and output types. There is no extra configuration needed on the task itself.
```typescript
import { hatchetWithMiddleware } from './client';
type TaskInput = {
message: string;
};
type TaskOutput = {
message: string;
};
export const taskWithMiddleware = hatchetWithMiddleware.task({
name: 'task-with-middleware',
fn: (input, ctx) => {
ctx.logger.info('task', { message: input.message }); // string (from TaskInput)
ctx.logger.info('task', { first: input.first }); // number (from GlobalInputType)
ctx.logger.info('task', { second: input.second }); // number (from GlobalInputType)
ctx.logger.info('task', { dependency: input.dependency }); // string (from Pre Middleware)
return {
message: input.message,
extra: 1,
};
},
});
// !!
```
The task's `input` type is the intersection of `TaskInput`, `GlobalInputType`, and the return type of the `before` middleware hook. The task's return type must satisfy `TaskOutput` and `GlobalOutputType`, while the caller receives the intersection of those with the `after` middleware return type.
#### Go
> **Info:** Middleware support for the Go SDK is coming soon. Join our [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
Pass a `deps` hash when defining a task. The resolved dependency values are available inside the task block via `ctx.deps`.
```ruby
ASYNC_TASK_WITH_DEPS = HATCHET.task(
name: "async_task_with_dependencies",
deps: {
sync_dep: sync_dep,
async_dep: async_dep,
sync_cm_dep: sync_cm_dep,
async_cm_dep: async_cm_dep,
chained_dep: chained_dep,
chained_async_dep: chained_async_dep
}
) do |input, ctx|
{
"sync_dep" => ctx.deps[:sync_dep],
"async_dep" => ctx.deps[:async_dep],
"async_cm_dep" => ctx.deps[:async_cm_dep],
"sync_cm_dep" => ctx.deps[:sync_cm_dep],
"chained_dep" => ctx.deps[:chained_dep],
"chained_async_dep" => ctx.deps[:chained_async_dep]
}
end
SYNC_TASK_WITH_DEPS = HATCHET.task(
name: "sync_task_with_dependencies",
deps: {
sync_dep: sync_dep,
async_dep: async_dep,
sync_cm_dep: sync_cm_dep,
async_cm_dep: async_cm_dep,
chained_dep: chained_dep,
chained_async_dep: chained_async_dep
}
) do |input, ctx|
{
"sync_dep" => ctx.deps[:sync_dep],
"async_dep" => ctx.deps[:async_dep],
"async_cm_dep" => ctx.deps[:async_cm_dep],
"sync_cm_dep" => ctx.deps[:sync_cm_dep],
"chained_dep" => ctx.deps[:chained_dep],
"chained_async_dep" => ctx.deps[:chained_async_dep]
}
end
DURABLE_ASYNC_TASK_WITH_DEPS = HATCHET.durable_task(
name: "durable_async_task_with_dependencies",
deps: {
sync_dep: sync_dep,
async_dep: async_dep,
sync_cm_dep: sync_cm_dep,
async_cm_dep: async_cm_dep,
chained_dep: chained_dep,
chained_async_dep: chained_async_dep
}
) do |input, ctx|
{
"sync_dep" => ctx.deps[:sync_dep],
"async_dep" => ctx.deps[:async_dep],
"async_cm_dep" => ctx.deps[:async_cm_dep],
"sync_cm_dep" => ctx.deps[:sync_cm_dep],
"chained_dep" => ctx.deps[:chained_dep],
"chained_async_dep" => ctx.deps[:chained_async_dep]
}
end
DURABLE_SYNC_TASK_WITH_DEPS = HATCHET.durable_task(
name: "durable_sync_task_with_dependencies",
deps: {
sync_dep: sync_dep,
async_dep: async_dep,
sync_cm_dep: sync_cm_dep,
async_cm_dep: async_cm_dep,
chained_dep: chained_dep,
chained_async_dep: chained_async_dep
}
) do |input, ctx|
{
"sync_dep" => ctx.deps[:sync_dep],
"async_dep" => ctx.deps[:async_dep],
"async_cm_dep" => ctx.deps[:async_cm_dep],
"sync_cm_dep" => ctx.deps[:sync_cm_dep],
"chained_dep" => ctx.deps[:chained_dep],
"chained_async_dep" => ctx.deps[:chained_async_dep]
}
end
DI_WORKFLOW = HATCHET.workflow(name: "dependency-injection-workflow")
# Workflow tasks with dependencies follow the same pattern
DI_WORKFLOW.task(:wf_task_with_dependencies) do |input, ctx|
{
"sync_dep" => SYNC_DEPENDENCY_VALUE,
"async_dep" => ASYNC_DEPENDENCY_VALUE
}
end
```
## Running a Worker
#### Python
No special worker configuration is needed — dependencies are evaluated automatically each time a task runs.
#### Typescript
Workers are created from the same middleware-enabled client. No special setup is required — the middleware hooks are applied automatically when tasks execute.
```typescript
import { hatchetWithMiddleware } from './client';
import { taskWithMiddleware } from './workflow';
async function main() {
const worker = await hatchetWithMiddleware.worker('task-with-middleware', {
workflows: [taskWithMiddleware],
});
await worker.start();
}
if (require.main === module) {
main();
}
```
#### Go
> **Info:** Middleware support for the Go SDK is coming soon. Join our [Discord](https://hatchet.run/discord) to stay up to date.
#### Ruby
No special worker configuration is needed — dependencies are resolved automatically before each task execution.
## Practical Examples
The examples below show TypeScript middleware for common production patterns. Each can be adapted to the Python dependency injection model by extracting the same logic into a dependency function.
### End-to-End Encryption
Encrypt sensitive input fields before they reach the Hatchet server, and decrypt the output on the way back. This ensures plaintext data never leaves your worker process.
```typescript
import { HatchetClient, HatchetMiddleware } from '@hatchet/v1';
import { randomUUID, createCipheriv, createDecipheriv, randomBytes } from 'crypto';
```
> **Info:** The `before` hook decrypts incoming data so your task function works with
> plaintext. The `after` hook encrypts the output before it is stored. The
> encryption key never leaves the worker environment.
### Offloading Large Payloads to S3
When task inputs or outputs exceed Hatchet's payload size limit (or you simply want to keep large blobs out of the control plane), upload them to S3 and pass a signed URL instead.
```typescript
import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
const ALGORITHM = 'aes-256-gcm';
const KEY = Buffer.from(process.env.ENCRYPTION_KEY!, 'hex');
type EncryptedEnvelope = { ciphertext: string; iv: string; tag: string };
function encrypt(plaintext: string): EncryptedEnvelope {
const iv = randomBytes(16);
const cipher = createCipheriv(ALGORITHM, KEY, iv);
const encrypted = Buffer.concat([cipher.update(plaintext, 'utf8'), cipher.final()]);
return {
ciphertext: encrypted.toString('base64'),
iv: iv.toString('base64'),
tag: cipher.getAuthTag().toString('base64'),
};
}
function decrypt(ciphertext: string, iv: string, tag: string): string {
const decipher = createDecipheriv(ALGORITHM, KEY, Buffer.from(iv, 'base64'));
decipher.setAuthTag(Buffer.from(tag, 'base64'));
return decipher.update(ciphertext, 'base64', 'utf8') + decipher.final('utf8');
}
type EncryptedInput = { encrypted?: EncryptedEnvelope };
const e2eEncryption: HatchetMiddleware = {
before: (input) => {
if (!input.encrypted) {
return input;
}
const { ciphertext, iv, tag } = input.encrypted;
const decrypted = JSON.parse(decrypt(ciphertext, iv, tag));
return { ...input, ...decrypted, encrypted: undefined };
},
after: (output) => {
const payload = JSON.stringify(output);
return { encrypted: encrypt(payload) };
},
};
const encryptionClient = HatchetClient.init().withMiddleware(e2eEncryption);
const s3 = new S3Client({ region: process.env.AWS_REGION });
const BUCKET = process.env.S3_BUCKET!;
const PAYLOAD_THRESHOLD = 256 * 1024; // 256 KB
async function uploadToS3(data: unknown): Promise {
const key = `hatchet-payloads/${randomUUID()}.json`;
await s3.send(
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: JSON.stringify(data),
ContentType: 'application/json',
})
);
return getSignedUrl(s3, new GetObjectCommand({ Bucket: BUCKET, Key: key }), {
expiresIn: 3600,
});
}
async function downloadFromS3(url: string): Promise {
const res = await fetch(url);
return res.json();
}
type S3Input = { s3Url?: string };
const s3Offload: HatchetMiddleware = {
before: async (input) => {
if (input.s3Url) {
const restored = (await downloadFromS3(input.s3Url)) as Record;
return { ...restored, s3Url: undefined };
}
return input;
},
after: async (output) => {
const serialized = JSON.stringify(output);
if (serialized.length > PAYLOAD_THRESHOLD) {
const url = await uploadToS3(output);
return { s3Url: url };
}
return output;
},
};
const s3Client = HatchetClient.init().withMiddleware(s3Offload);
```
The caller is responsible for uploading oversized inputs to S3 before triggering the task. The `before` hook only handles the download side. You can use the same `uploadToS3` helper on the caller side to upload the input and pass `{ __s3Url: url }` as the task input.
## FAQ
### What is Hatchet middleware and how does it differ from Express middleware?
Hatchet middleware runs **inside the worker process** around each task invocation — not on an HTTP request path. A `before` hook transforms input before the task runs, and an `after` hook transforms output after. Unlike Express middleware, there is no `next()` function; hooks return their result directly and the runner chains them automatically.
### Can I use middleware with both tasks and workflows?
Yes. Middleware is registered on the `HatchetClient` instance, so it applies to every task created from that client — whether the task is a standalone `client.task()` or part of a multi-step `client.workflow()`. Each step in a workflow will have middleware applied independently.
### Does middleware run on the server or on the worker?
Middleware runs entirely **on the worker**. The Hatchet server never sees or executes your middleware code. This is what makes patterns like end-to-end encryption possible — plaintext data stays within your infrastructure.
### What happens if my middleware throws an error?
If a `before` or `after` hook throws (or returns a rejected `Promise`), the task run fails with that error. There is no automatic retry of middleware itself, but the task's configured retry policy will still apply, re-running the task (and its middleware) from scratch.
### Can I use async/await in middleware hooks?
Yes. Both `before` and `after` hooks can be synchronous or asynchronous. If a hook returns a `Promise`, the worker will `await` it before proceeding to the next hook or the task function.
### How do I share state between `before` and `after` hooks?
The `after` hook receives the task input (after `before` hooks have run) as its third argument. Add fields in `before` (e.g. `startedAt`, `traceId`) and read them from `input` in `after`. There is no separate shared context object — the input itself is the carrier.
### Does middleware apply to child tasks spawned via fanout?
Middleware is scoped to the **client instance**. If a child task is defined on the same middleware-enabled client, its middleware will run when that child task executes. If the child task uses a different client instance, only that client's middleware (if any) applies.
### Can I selectively skip middleware for certain tasks?
Middleware applies to **all** tasks on a given client. To skip middleware for specific tasks, create a second client without middleware and define those tasks on it. This is a deliberate design choice — middleware is a cross-cutting concern, and selective opt-out is handled at the client boundary.
### Is there a performance overhead to using middleware?
Middleware hooks are plain JavaScript functions that run in-process on the worker. The overhead is the execution time of your hook code. For lightweight operations (adding a field, logging), the overhead is negligible. For heavier operations (network calls like S3 uploads or decryption), the task's total duration will include that time, so keep hooks as efficient as possible.
### What is the difference between global types and middleware types in TypeScript?
Global types (`HatchetClient.init()`) define fields that **callers must provide** when triggering a task. Middleware types (inferred from `withMiddleware` return values) define fields that are **injected at runtime** by the worker. Both end up on the task's `input` type, but only global types appear in the caller-facing `run()` signature.
### Can I use middleware for rate limiting or authentication?
Yes. A `before` hook can check rate limits, validate API keys, or verify JWTs before the task runs. If the check fails, throw an error to abort the task. However, for rate limiting specifically, consider using Hatchet's built-in [rate limiting](/home/rate-limits) feature, which operates at the scheduling layer and is more efficient than in-worker checks.
### How do I test middleware in isolation?
Middleware hooks are plain functions — you can unit-test them directly by calling them with mock input and a mock context object. For integration tests, the e2e test pattern of creating a client, attaching middleware, defining a task, starting a worker, and asserting on the result works well. See the [middleware example on GitHub](https://github.com/hatchet-dev/hatchet/tree/main/examples/typescript/middleware) for a complete test setup.
---
# Streaming in Hatchet
Hatchet tasks can stream data back to a consumer in real-time. This has a number of valuable uses, such as streaming the results of an LLM call back from a Hatchet worker to a frontend or sending progress updates as a task chugs along.
## Publishing Stream Events
You can stream data out of a task run by using the `put_stream` (or equivalent) method on the `Context`.
#### Python
```python
anna_karenina = """
Happy families are all alike; every unhappy family is unhappy in its own way.
Everything was in confusion in the Oblonskys' house. The wife had discovered that the husband was carrying on an intrigue with a French girl, who had been a governess in their family, and she had announced to her husband that she could not go on living in the same house with him.
"""
def create_chunks(content: str, n: int) -> Generator[str, None, None]:
for i in range(0, len(content), n):
yield content[i : i + n]
chunks = list(create_chunks(anna_karenina, 10))
@hatchet.task()
async def stream_task(input: EmptyModel, ctx: Context) -> None:
# 👀 Sleeping to avoid race conditions
await asyncio.sleep(2)
for chunk in chunks:
await ctx.aio_put_stream(chunk)
await asyncio.sleep(0.20)
```
#### Typescript
```typescript
const annaKarenina = `
Happy families are all alike; every unhappy family is unhappy in its own way.
Everything was in confusion in the Oblonskys' house. The wife had discovered that the husband was carrying on an intrigue with a French girl, who had been a governess in their family, and she had announced to her husband that she could not go on living in the same house with him.
`;
function* createChunks(content: string, n: number): Generator {
for (let i = 0; i < content.length; i += n) {
yield content.slice(i, i + n);
}
}
export const streamingTask = hatchet.task({
name: 'stream-example',
fn: async (_, ctx) => {
await sleep(2000);
for (const chunk of createChunks(annaKarenina, 10)) {
ctx.putStream(chunk);
await sleep(200);
}
},
});
```
#### Go
```go
const annaKarenina = `
Happy families are all alike; every unhappy family is unhappy in its own way.
Everything was in confusion in the Oblonskys' house. The wife had discovered that the husband was carrying on an intrigue with a French girl, who had been a governess in their family, and she had announced to her husband that she could not go on living in the same house with him.
`
func createChunks(content string, n int) []string {
var chunks []string
for i := 0; i < len(content); i += n {
end := i + n
if end > len(content) {
end = len(content)
}
chunks = append(chunks, content[i:end])
}
return chunks
}
func StreamTask(ctx hatchet.Context, input StreamTaskInput) (*StreamTaskOutput, error) {
time.Sleep(2 * time.Second)
chunks := createChunks(annaKarenina, 10)
for _, chunk := range chunks {
ctx.PutStream(chunk)
time.Sleep(200 * time.Millisecond)
}
return &StreamTaskOutput{
Message: "Streaming completed",
}, nil
}
```
#### Ruby
```ruby
ANNA_KARENINA = <<~TEXT
Happy families are all alike; every unhappy family is unhappy in its own way.
Everything was in confusion in the Oblonskys' house. The wife had discovered that the husband was carrying on an intrigue with a French girl, who had been a governess in their family, and she had announced to her husband that she could not go on living in the same house with him.
TEXT
STREAM_CHUNKS = ANNA_KARENINA.scan(/.{1,10}/)
STREAM_TASK = HATCHET.task(name: "stream_task") do |input, ctx|
# Sleeping to avoid race conditions
sleep 2
STREAM_CHUNKS.each do |chunk|
ctx.put_stream(chunk)
sleep 0.20
end
end
```
This task will stream small chunks of content through Hatchet, which can then be consumed elsewhere. Here we use some text as an example, but this is intended to replicate streaming the results of an LLM call back to a consumer.
## Consuming Streams
You can easily consume stream events by using the stream method on the workflow run ref that the various [fire-and-forget](/v1/running-your-task#fire-and-forget) methods return.
#### Python
```python
ref = await stream_task.aio_run(wait_for_result=False)
async for chunk in hatchet.runs.subscribe_to_stream(ref.workflow_run_id):
print(chunk, flush=True, end="")
```
#### Typescript
```typescript
const ref = await streamingTask.runNoWait({});
const id = await ref.getWorkflowRunId();
for await (const content of hatchet.runs.subscribeToStream(id)) {
process.stdout.write(content);
}
```
#### Go
```go
func main() {
client, err := hatchet.NewClient()
if err != nil {
log.Fatalf("Failed to create Hatchet client: %v", err)
}
ctx := context.Background()
streamingWorkflow := shared.StreamingWorkflow(client)
workflowRun, err := streamingWorkflow.RunNoWait(ctx, shared.StreamTaskInput{})
if err != nil {
log.Fatalf("Failed to run workflow: %v", err)
}
id := workflowRun.RunId
stream := client.Runs().SubscribeToStream(ctx, id)
for content := range stream {
fmt.Print(content)
}
fmt.Println("\nStreaming completed!")
}
```
#### Ruby
```ruby
ref = STREAM_TASK.run_no_wait
HATCHET.runs.subscribe_to_stream(ref.workflow_run_id) do |chunk|
print chunk
end
```
In the examples above, this will result in the famous text below being gradually printed to the console, bit by bit.
```
Happy families are all alike; every unhappy family is unhappy in its own way.
Everything was in confusion in the Oblonskys' house. The wife had discovered that the husband was carrying on an intrigue with a French girl, who had been a governess in their family, and she had announced to her husband that she could not go on living in the same house with him.
```
You must begin consuming the stream before any events are published. Any
events published before a consumer is initialized will be dropped. In
practice, this will not be an issue in most cases, but adding a short sleep
before beginning streaming results back can help.
## Streaming to a Web Application
It's common to want to stream events out of a Hatchet task and back to the frontend of your application, for consumption by an end user. As mentioned before, some clear cases where this is useful would be for streaming back progress of some long-running task for a customer to monitor, or streaming back the results of an LLM call.
In both cases, we recommend using your application's backend as a proxy for the stream, where you would subscribe to the stream of events from Hatchet, and then stream events through to the frontend as they're received by the backend.
#### Python
For example, with FastAPI, you'd do the following:
```python
hatchet = Hatchet()
app = FastAPI()
@app.get("/stream")
async def stream() -> StreamingResponse:
ref = await stream_task.aio_run(wait_for_result=False)
return StreamingResponse(
hatchet.runs.subscribe_to_stream(ref.workflow_run_id), media_type="text/plain"
)
```
#### Typescript
For example, with NextJS backend-as-frontend, you'd do the following:
```typescript
export async function GET(): Promise {
try {
const ref = await streamingTask.runNoWait({});
const workflowRunId = await ref.getWorkflowRunId();
const stream = Readable.from(hatchet.runs.subscribeToStream(workflowRunId));
return new Response(Readable.toWeb(stream), {
headers: {
'Content-Type': 'text/plain',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
},
});
} catch (error) {
return new Response('Internal Server Error', { status: 500 });
}
}
```
#### Go
For example, with Go's built-in HTTP server, you'd do the following:
```go
func main() {
client, err := hatchet.NewClient()
if err != nil {
log.Fatalf("Failed to create Hatchet client: %v", err)
}
streamingWorkflow := shared.StreamingWorkflow(client)
http.HandleFunc("/stream", func(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
w.Header().Set("Content-Type", "text/plain")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
workflowRun, err := streamingWorkflow.RunNoWait(ctx, shared.StreamTaskInput{})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
stream := client.Runs().SubscribeToStream(ctx, workflowRun.RunId)
flusher, _ := w.(http.Flusher)
for content := range stream {
fmt.Fprint(w, content)
if flusher != nil {
flusher.Flush()
}
}
})
server := &http.Server{
Addr: ":8000",
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
}
if err := server.ListenAndServe(); err != nil {
log.Println("Failed to start server:", err)
}
}
```
#### Ruby
Then, assuming you run the server on port `8000`, running `curl -N http://localhost:8000/stream` would result in the text streaming back to your console from Hatchet through your FastAPI proxy.
---
# Managing Environments with Hatchet
## Multiple Developers, One Orchestrator
When multiple developers share a single Hatchet orchestrator, conflicts can arise as workflow runs and events intermingle. Without proper isolation, one developer's workflows might interfere with another's testing or development work.
Hatchet provides three key solutions for managing this challenge: namespaces, multi-tenancy, and a local Hatchet instance.
### Solution 1: Multi-Tenancy
The easiest way to isolate environments for different developers or teams is to use Hatchet's multi-tenancy feature. Each tenant represents a separate environment with its own set of workflows and workers. To add a new tenant for each developer, create an organization and follow these steps:
1. Access the organization dropdown in the dashboard (top right)
2. Select the `+` icon next to your organization's name
3. Generate a new token for that tenant
4. Each developer configures their environment with their designated tenant token
### Solution 2: Local Hatchet Instance
If you are using Hatchet locally, you can create a local instance of Hatchet to manage your isolated local development environment. Follow instructions [here](/self-hosting/hatchet-lite) to get started.
---
# Troubleshooting Hatchet Workers
This guide covers common issues when deploying and operating Hatchet workers.
## Quick debugging checklist
Before diving into specific issues, run through these checks:
1. **Verify your API token** — make sure `HATCHET_CLIENT_TOKEN` matches the token generated in the Hatchet dashboard for your tenant.
2. **Check worker logs** — look for connection errors, heartbeat failures, or crash traces in your worker output.
3. **Check the dashboard** — navigate to the Workers tab to see if your worker is registered and healthy.
4. **Confirm network connectivity** — workers need to reach the Hatchet engine over gRPC. Firewalls, VPNs, or missing TLS configuration can block this.
5. **Check SDK version** — ensure your SDK version is compatible with your engine version. Mismatches can cause subtle failures.
## Could not send task to worker
If you see this error in the event history of a task, it could mean several things:
1. The worker is closing its network connection while the task is being sent. This could be caused by the worker crashing or going offline.
2. The payload is too large for the worker to accept or the Hatchet engine to send. The default maximum payload size is 4MB. Consider reducing the size of the input data or output data of your tasks.
3. The worker has a large backlog of tasks in-flight on the network connection and is rejecting new tasks. This can occur if workers are geographically distant from the Hatchet engine or if there are network issues causing delays. Hatchet Cloud runs by default in `us-west-2` (Oregon, USA), so consider deploying your workers in a region close to that for the best performance.
If you are self-hosting, you can increase the maximum backlog size via the `SERVER_GRPC_WORKER_STREAM_MAX_BACKLOG_SIZE` environment variable in your Hatchet engine configuration. The default is 20.
## No workers visible in dashboard
If you have deployed workers but they are not visible in the Hatchet dashboard, it is likely that:
1. Your API token is invalid or incorrect. Ensure that the token you are using to start the worker matches the token generated in the Hatchet dashboard for your tenant.
2. Worker heartbeats are not reaching the Hatchet engine. You will see noisy logs in the worker output if this is the case.
## Tasks stuck in QUEUED state
If tasks remain in the `QUEUED` state and never move to `RUNNING`:
1. **No workers registered for the task** — check the Workers tab in the dashboard and confirm a worker is registered that handles the task name. If you recently renamed a task, make sure the worker has been restarted with the updated code.
2. **All worker slots are full** — if every slot is occupied by other tasks, new tasks will wait in the queue. Check worker utilization in the dashboard or increase the [slot count](/v1/workers#slots). A task with a [slot cost](/v1/advanced-assignment/slot-cost) higher than every worker's slot count can never be scheduled, and is cancelled when its schedule timeout is reached.
3. **Concurrency or rate limit is blocking** — if you've configured [concurrency limits](/v1/concurrency) or [rate limits](/v1/rate-limits), tasks may be held back intentionally. Review your configuration.
## Worker keeps disconnecting
If your worker repeatedly connects and then drops:
1. **Resource exhaustion** — the worker process may be running out of memory or CPU and getting killed by the OS or orchestrator (OOM kill). Check system logs and increase resource limits.
2. **Network instability** — intermittent connectivity between the worker and the Hatchet engine will cause reconnection cycles. Check for packet loss or high latency between the worker and the engine.
3. **Graceful shutdown not configured** — if your deployment platform sends `SIGTERM` and the worker doesn't handle it, in-flight tasks may be interrupted. Ensure your worker handles shutdown signals and gives tasks time to complete.
## Phantom workers active in dashboard
This is often due to workers still running in your deployed environment. We see this most often with very long termination periods for workers, or in local development environments where worker processes are leaking. If you are in a local development environment, you can usually view running Hatchet worker processes via `ps -a | grep worker` (or whatever your entrypoint binary is called) and kill them manually.
---
# Single-Sign-On
> **Info:** Single Sign On is only available through our Cloud Scale plan. See
> [pricing](https://hatchet.run/pricing) for more details.
In addition to Google and Github OAuth, we also provide a custom SSO solution for enterprises that wish
to use their own Identity Provider (IdP).
## Supported IdPs
- Okta
- Microsoft Entra
- OneLogin
- JumpCloud
- Google
- generic OIDC providers
## Setting up SSO
### IdP Configuration
The first step is setting up the SSO configuration. This can be done by clicking on "Set up SSO" in Settings > Organization > SSO
and filling out the fields provided. The fields vary by platform, but for all platforms you must copy the "Redirect / Callback" URL and set it as the callback
URL in your IdP.
### Domain Verification
To allow a user to sign in with SSO, you must also link your organization's email domain to your SSO configuration.
This is done using DNS verification--you must add the `hatchet-sso-verify` to your domain's DNS TXT records with a verification
token to link the domain. Once the domain is verified, all users who sign in with SSO using that email domain will use your SSO
configuration to authenticate. There may be a short delay between updating the TXT record and verification, as the records
need to propagate. Here is an example verification record:
Type, Domain Name, Record
TXT, hatchet.run, "hatchet-sso-verify=b2b92da1-4314-11f1-a43b-5eff43bccba3"
### Forcing SSO
By enabling the Force SSO toggle, all users will be forced to sign in with SSO, disabling all other login methods.
> **Warning:** Only enable forced SSO after you have confirmed that SSO is working correctly,
> otherwise you may be locked out of your organization.
> Contact support if you end up in this
> state.
---
# User Groups
> **Info:** User Groups are only available through our Cloud Scale plan. See
> [pricing](https://hatchet.run/pricing) for more details.
User groups make granting automatic access to tenants within an organization easier.
### Creating a user group
Navigate to Settings > Team, then click on the New Group button.
In the modal, input the name for the group, the tenant role that users synced with that group will have, and tags to determine which tenant the user group will be synced to.
### Adding tags to tenants
Navigate to the tenants page in Settings, and select "Edit tags" from the dropdown for a tenant.
Add the desired tags.
Tags can also be set when creating a new tenant.
## Tag Syncing
User groups work by automatically granting access for the users inside the group to tenants that have a _subset_
of the user group's tags. The diagram below shows an example tenant-user group setup. Three tenants — "Preview" (tagged `production` and `staging`), "Production" (tagged
`production`), and "Staging" (tagged `staging`) — alongside three user groups with the same tag combinations. Each
tenant lists the users who are synced into it automatically, based on which groups' tags are a superset of its own:
```mermaid
%%{init: {'themeVariables': {'fontSize': '13px'}}}%%
flowchart LR
subgraph UG["User Groups"]
direction TB
G1("Everyone tags: production, staging a@example.com ADMIN")
G2("Production Team tags: production c@example.com MEMBER")
G3("Staging Team tags: staging b@example.com MEMBER")
end
subgraph TN["Tenants"]
direction TB
T1("Preview tags: production, staging a@example.com ADMIN")
T2("Production tags: production a@example.com ADMIN c@example.com MEMBER")
T3("Staging tags: staging a@example.com ADMIN b@example.com MEMBER")
end
G1 --> T1
G1 --> T2
G1 --> T3
G2 --> T2
G3 --> T3
style G1 stroke-dasharray: 4 3,fill:#f3e5f5,stroke:#8e24aa,color:#4a148c
style G2 stroke-dasharray: 4 3,fill:#e3f2fd,stroke:#1976d2,color:#0d47a1
style G3 stroke-dasharray: 4 3,fill:#fff3e0,stroke:#f57c00,color:#e65100
style T1 fill:#f3e5f5,stroke:#8e24aa,stroke-width:2px,color:#4a148c
style T2 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#0d47a1
style T3 fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#e65100
style UG fill:transparent,stroke:#8a8a8a,stroke-width:1px
style TN fill:transparent,stroke:#8a8a8a,stroke-width:1px
```
## Organization Owners
Organization owners are exempt from the tag-syncing roles, they are added to every tenant in the organization with role "OWNER".
---
# Architecture & Guarantees
This page explains how Hatchet is put together, what the main components do, and what reliability guarantees you should design your workers around.
## Architecture overview
Hatchet has three main moving pieces:
- **API server**: the HTTP surface area for triggering workflows, querying state, and powering the UI
- **Engine**: schedules and dispatches work, enforces dependencies/policies, and records state transitions durably
- **Workers**: your processes that run the actual task code
State is stored durably (PostgreSQL is the source of truth). In many deployments that’s enough—no separate broker required—while self-hosted high-throughput setups can add additional components (for example, RabbitMQ) based on your needs.
Hatchet Cloud and self-hosted Hatchet share the same architecture; the difference is who runs and operates the Hatchet services.
```mermaid
graph LR
subgraph "External (Optional)"
EXT[Webhooks Events]
end
subgraph "Your Infrastructure"
APP[Your API, App, Service, etc.]
W[Workers]
end
subgraph "Hatchet"
API[API Server]
ENG[Engine]
DB[(Database)]
end
EXT --> API
APP <--> API
API --> ENG
ENG <--> DB
API <--> DB
ENG <-.->|gRPC| W
classDef userInfra fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#0d47a1
classDef hatchet fill:#f1f8e9,stroke:#388e3c,stroke-width:2px,color:#1b5e20
classDef external fill:#fff8e1,stroke:#f57c00,stroke-width:2px,color:#e65100
class APP,W userInfra
class API,ENG,DB hatchet
class EXT external
```
## Core components
### API server
The API server is the front door to Hatchet. It’s what your application and the Hatchet UI talk to in order to:
- trigger workflows with input data
- query workflow/task state (and, where supported, subscribe to updates)
- manage resources like schedules and settings
- ingest webhooks/events (optional)
### Engine
The engine is responsible for turning “a workflow should run” into “these tasks are ready and should be executed.” In practice, it:
- evaluates workflow dependencies
- enforces policies like concurrency limits, rate limits, and priorities
- schedules ready tasks and dispatches them to connected workers
- records state transitions durably and applies retry/timeout behavior
- runs scheduled/cron workflows
Workers connect to the engine over bidirectional gRPC, which allows low-latency dispatch and frequent status updates.
### Workers
Workers are your processes. They connect to the engine, receive tasks, run your code, and report progress/results back to Hatchet.
Workers are intentionally flexible: you can run them locally, in containers, or on VMs, and you can scale workers independently from the Hatchet services. You can also run different “types” of workers (and even different languages) depending on what your system needs.
### Storage (and optional messaging)
PostgreSQL is the durable store for workflow definitions and execution state (queued/running/completed, inputs/outputs, retries, etc.). In self-hosted deployments, you can start with PostgreSQL-only and add components like RabbitMQ if you need higher throughput.
## Guarantees & tradeoffs
Hatchet aims to sit in the middle: more structure than a simple queue, but simpler to operate than a full distributed workflow system.
### Good fit for
- **Workflow orchestration** with dependencies, retries, and timeouts
- **Durable background jobs** where “don’t lose work” matters
- **Moderate to high throughput** systems (and a path to higher scale with tuning/sharding). If you’re pushing the limits, [contact us](mailto:support@hatchet.run).
- **Multi-language / polyglot workers**
- **Teams already on PostgreSQL** who want operational simplicity
- **Cloud or air-gapped environments** ([Hatchet Cloud](https://cloud.hatchet.run) or [self-hosting](/self-hosting))
### Not a good fit for
- **Extremely high throughput** without sharding/custom tuning (for example, sustaining 10,000+ tasks/sec)
- **Sub-millisecond dispatch latency** requirements
- **In-memory-only queuing** where durability is unnecessary
- **Serverless-only runtimes** (e.g. AWS Lambda / Cloud Functions) as your primary worker model
## Core reliability guarantees
### At-least-once task execution
Hatchet is **at least once**: tasks are not silently dropped, and failures retry according to your configuration. This also means **a task can run more than once**, so your task code should be **idempotent** (or otherwise safe to retry).
### Durable state transitions
Workflow state is persisted in PostgreSQL, and state transitions are performed transactionally. This helps keep dependency resolution consistent and makes the system resilient to restarts and transient failures.
### Execution policies are explicit
By default, task assignment is FIFO, and you can change execution behavior using:
- [Concurrency policies](/v1/concurrency)
- [Rate limits](/v1/rate-limits)
- [Priorities](/v1/priority)
### Stateless services; resilient workers
The engine and API server are designed to restart without losing state, which also enables horizontal scaling by running multiple instances. Workers reconnect after network interruptions and can run close to your services (or close to Hatchet) depending on your latency goals.
## Performance expectations
Real-world performance depends heavily on topology (worker ↔ engine network latency), database sizing, and workload shape.
- **Dispatch latency**: often sub-50ms with PostgreSQL-backed storage; in optimized, “hot worker” setups it can be closer to ~25ms P95.
- **Throughput**: varies by setup. PostgreSQL-only deployments often handle hundreds of tasks/sec per engine instance; higher throughput typically requires additional tuning and/or components like RabbitMQ. With tuning and sharding, Hatchet can scale into the high tens of thousands of tasks/sec—[contact us](mailto:support@hatchet.run) if you want to design for that.
- **Common bottlenecks**: DB connection limits, large payloads (e.g. > 1MB), complex dependency graphs, and cross-region latency.
> **Warning:** **Not seeing expected performance?**
>
> If you're not seeing the performance you expect, please [reach out to us](https://cal.com/team/hatchet/talk-to-us) or [join our community](https://hatchet.run/discord) to explore tuning options.
## Next Steps
- **[Quick Start](/v1/quickstart)**: set up your first Hatchet worker
- **[Self-Hosting](/self-hosting)**: deploy Hatchet on your own infrastructure
---
# Cloud vs OSS
Hatchet is available as **Hatchet Cloud** (managed) and as **open source** (self-hosted). The programming model is the same: you write tasks/workflows in code and run workers that connect to Hatchet.
This page helps you decide which deployment model fits your team.
## Quick decision guide
Choose **Hatchet Cloud** if you want:
- the Hatchet control plane operated for you (upgrades, scaling, backups)
- the fastest path to production
- a status page and managed incident response
Choose **self-hosted (OSS)** if you need:
- full control over infrastructure and networking
- strict data residency or air-gapped environments
- a deployment you can customize and operate with your own tooling
## What’s the same in both
- **SDK + worker model**: your workers run your code and connect to Hatchet
- **Durability + retries**: tasks are durably tracked and retry according to configuration
- **Observability surfaces**: you can inspect runs, workers, and workflow history
- **Core semantics**: the same workflows/tasks/concurrency patterns apply
## What changes (operational responsibilities)
### Hatchet Cloud (managed)
Hatchet runs and operates the Hatchet services. You bring:
- your worker processes
- your application code that triggers workflows
- your operational policies (timeouts, retries, concurrency, rate limits)
For security and compliance documentation, see the **[Hatchet Trust Center](https://trust.hatchet.run/)**. For current incidents and historical uptime, see **[status.hatchet.run](https://status.hatchet.run/)**.
### Self-hosted (OSS)
You run and operate the Hatchet services and their dependencies. Typical responsibilities include:
- provisioning and scaling the Hatchet services
- managing PostgreSQL (and any optional components you deploy)
- backups, upgrades, and monitoring
- network security and access control for the API/DB
If you’re planning production usage, start with:
- [Self Hosting](/self-hosting)
- [High Availability](/self-hosting/high-availability)
- [Security](/v1/security)
## Migrating between Cloud and self-hosted
You can move between deployment models without rewriting worker code. In practice, migration usually means:
- pointing workers and clients at a new endpoint
- swapping credentials/tokens
- validating environment-specific settings (TLS, networking, retention, etc.)
## Next steps
- **[Quickstart](/v1/quickstart)**: run a worker and trigger your first workflow
- **[Architecture & Guarantees](/v1/architecture-and-guarantees)**: understand reliability semantics and tradeoffs
- **[Self Hosting](/self-hosting)**: deploy Hatchet on your own infrastructure
---
# Security
This page points you to Hatchet's security resources and highlights the most important security considerations for Hatchet Cloud and self-hosted deployments.
## Trust center
Hatchet is SOC 2 Type II, HIPAA, and GDPR compliant. Company-level security practices, compliance reports, and security documentation are available at the **[Hatchet Trust Center](https://trust.hatchet.run/)**.
## Same source, same security
Hatchet Cloud and self-hosted Hatchet run the same codebase. The open source project is 100% MIT licensed and undergoes regular third-party penetration testing. Findings are remediated across both deployment models, so security improvements benefit all users equally.
## Hatchet Cloud
Hatchet Cloud is Hatchet's managed service:
- **Encryption in transit**: all API and worker traffic is encrypted with TLS. gRPC connections between workers and the engine use TLS by default.
- **Encryption at rest**: data stored in Hatchet Cloud is encrypted at rest.
- **Tenant isolation**: each tenant's data is logically isolated. Requests are authenticated and scoped to a single tenant.
- **Authentication**: API tokens are scoped per-tenant with configurable expiration. The dashboard supports SSO via Google, GitHub, and more coming soon.
- **Penetration testing**: Hatchet Cloud is regularly tested by independent security firms. Findings are tracked and remediated on a defined timeline.
- **Infrastructure**: Hatchet Cloud runs on AWS with private networking, automated patching, and centralized logging.
For the definitive controls, policies, and compliance reports, refer to the **[Hatchet Trust Center](https://trust.hatchet.run/)**.
## Self-hosted
When you self-host Hatchet, your security posture depends on how you deploy and operate the Hatchet services and their dependencies. A practical baseline:
- **Put TLS in front of the API**: terminate TLS at your ingress/load balancer (or directly on the API) and only expose it to the networks that need it.
- **Treat tokens and DB credentials as secrets**: use a secrets manager and rotate credentials; avoid committing secrets into git or baking them into images.
- **Limit network reachability**: restrict access to the Hatchet API and PostgreSQL to trusted networks (VPC, private subnets, or Kubernetes network policies).
- **Use least privilege**: run Hatchet with the minimum DB permissions needed; don't reuse "admin" DB credentials.
- **Stay current**: keep Hatchet and dependencies up to date to pick up security fixes.
See [Self Hosting](/self-hosting) for deployment and configuration guidance, or [contact us](mailto:support@hatchet.run) for help.
---
# Audit Logs
Hatchet records audit logs for key actions performed across your organization, giving you visibility into who did what, when, and from where.
> **Info:** Audit logs are available on **Business** plans and above. If you're on the
> open-source edition and need audit logs, [contact
> us](mailto:support@hatchet.run) to learn more about upgrading.
## What gets logged
Every audit log entry captures the following:
Field, Description
**Actor**, The user or API key that performed the action
**Action**, The operation performed (e.g. `ApiTokenCreate`, `TenantMemberDelete`)
**Resource type**, The type of resource acted upon (e.g. `workflow-run`, `api-token`)
**Resource ID**, The specific resource that was affected
**IP address**, The IP address of the actor (HTTP requests only)
**User agent**, The user agent string of the request (HTTP requests only)
**Timestamp**, When the action occurred
**Correlation ID**, An optional ID for grouping related actions together (gRPC requests)
## Audited actions
The following actions are currently recorded as audit log entries:
Action, Resource Type, Description
`TenantInviteAccept`, `tenant-invite`, A user accepts a tenant invitation
`TenantMemberDelete`, `tenant-member`, A tenant member is removed
`ApiTokenCreate`, `api-token`, A new API token is created
`ApiTokenUpdateRevoke`, `api-token`, An API token is revoked
`V1WorkflowRunCreate`, `workflow-run`, A workflow run is triggered via the API
`ScheduledWorkflowRunCreate`, `scheduled-workflow`, A scheduled workflow run is created
## Actor types
Audit log entries distinguish between two types of actors:
- **User** — actions performed by a logged-in user through the dashboard or API. These entries include the actor's IP address and user agent.
- **API key** — actions performed programmatically via an API key (e.g. triggering workflow runs over gRPC). These entries may include a correlation ID for grouping related actions.
## Retention
Audit log entries are retained for **30 days**. Entries older than 30 days are automatically removed.
## Viewing audit logs
Organization admins can view audit logs in the Hatchet dashboard under the organization settings. Logs can be filtered by tenant and time range.
## API access
Audit logs can also be retrieved programmatically via the Management API:
```
GET /api/v1/management/organizations/{organization}/audit-logs
```
Query parameters:
Parameter, Type, Default, Description
`tenant`, UUID, all active tenants in the organization, Filter logs to a specific tenant
`limit`, integer, `1000`, Maximum number of results to return
`offset`, integer, `0`, Number of results to skip
`since`, ISO 8601, 24 hours ago, Start of the time range
`until`, ISO 8601, now, End of the time range
Results are ordered by timestamp descending (most recent first).
---
# Region availability
Hatchet Cloud is available in multiple regions so you can run workloads close to your users and data.
## Current regions
**Hatchet Cloud** ([cloud.hatchet.run](https://cloud.hatchet.run)) Control Plane is deployed in **aws-us-west-2** (Oregon).
We are expanding Data Plane availability. Planned or available regions include:
Region, Location, Status
aws-us-west-2, Oregon (US), **Live**
aws-us-east-1, N. Virginia (US), **Live**
aws-eu-west-1, Ireland, **Live**
aws-ap-southeast-2, Sydney, Private Beta
## Request a region
We are always open to rolling out new regions based on demand. If you need a specific region for latency or compliance, [contact us](mailto:support@hatchet.run) and we can discuss availability.
---
# Uptime and status
For Hatchet Cloud availability and incident updates, use the status page. For self-hosted deployments, availability depends on your own infrastructure.
## Hatchet Cloud status page
Use **[status.hatchet.run](https://status.hatchet.run/)** for real-time status and incident history for Hatchet Cloud and related services:
- **API**: Hatchet API availability
- **Hatchet Cloud**: `cloud.hatchet.run`
- **Website**: `hatchet.run` and documentation sites
You can also subscribe to updates (email/SMS/etc.) directly from the status page.
## Self-hosted deployments
If you self-host Hatchet, you’re responsible for uptime, monitoring, backups, and upgrade procedures.
- **Deployment guidance**: [Self Hosting](/self-hosting)
- **Redundancy & failover**: [High Availability](/self-hosting/high-availability)
---
# Developer experience
Hatchet is designed to be practical day-to-day: write workflows in code, run workers locally with a tight feedback loop, and debug production runs with good visibility.
## Workflows as code
You define tasks and workflows in your application code, then trigger them with input data. Hatchet handles the operational pieces you’d otherwise build yourself:
- **Durability** (work isn’t lost on restarts)
- **Retries/timeouts**
- **Concurrency and rate limiting**
- **Visibility into what ran, where, and why**
## Dashboard (UI)
The dashboard is where you go to understand “what is happening right now?”:
- **Runs**: status, inputs/outputs, and execution history
- **Workers**: connected workers and health
- **Workflows**: definitions and recent activity
- **Settings**: tenants, API tokens, configuration
It’s useful for debugging, operational checks, and ad-hoc triggers.
## CLI
The [Hatchet CLI](/reference/cli) is the fastest way to develop and operate Hatchet from your terminal:
- **`hatchet worker dev`**: run a local worker with hot reload
- **`hatchet trigger`**: trigger a workflow from the command line (handy for smoke tests)
- **`hatchet tui`**: terminal UI for runs/workers/workflows
- **`hatchet profile`**: switch between tenants and environments
See the [CLI reference](/reference/cli) for installation and the full command set.
## Coding agents (MCP)
If you use AI coding tools in your editor, Hatchet’s docs can be used via an [MCP (Model Context Protocol) server](/v1/using-coding-agents). We also publish “agent skills” (short, step-by-step playbooks) so coding agents can run common Hatchet workflows—like starting a worker, triggering a workflow, and debugging a run—without guessing at CLI usage.
See [Using Coding Agents](/v1/using-coding-agents) for setup.
---
# Migrating from Celery to Hatchet
Celery is a mature and widely used background task system. This guide assumes you have already decided to migrate a Python project from Celery to Hatchet and want to understand what code and configuration need to change. Each section starts with a common Celery pattern, then shows what replaces it in Hatchet and what to watch out for.
> **Info:** This guide covers common Celery patterns. It does not cover every Celery
> setting or attempt a full infrastructure migration. For a discussion of
> Celery's operational trade-offs, see [this blog
> post](https://hatchet.run/blog/problems-with-celery).
## Migration pattern lookup
Use this table to find the Celery pattern you have in your project and jump to the section that shows the Hatchet replacement. The table is a lookup aid; the sections below explain the migration details, caveats, and code changes.
Step, Celery, Hatchet replacement, Migration category
[1](#step-1-update-dependencies-and-runtime-configuration), `celery[...]` + broker dependencies, `hatchet-sdk`, Dependency change
[1](#step-1-update-dependencies-and-runtime-configuration), Celery config / env vars, `HATCHET_CLIENT_TOKEN`, Operational change
[1](#step-1-update-dependencies-and-runtime-configuration), Worker / Beat / Flower processes, Hatchet worker + engine/cloud, Operational change
[2](#step-2-replace-the-celery-app-with-a-hatchet-client), `Celery("app", broker=..., backend=...)`, `Hatchet()` + `HATCHET_CLIENT_TOKEN`, Operational change
[3](#step-3-convert-task-definitions), `@app.task` / `@shared_task`, `@hatchet.task()`, Small rewrite
[3](#step-3-convert-task-definitions), `def my_task(arg1, arg2)`, `def my_task(input: MyInput, ctx: Context)`, Small rewrite
[4](#step-4-invoke-tasks-with-input-models), `task.delay(...)` / `.apply_async(...)`, `.run(..., wait_for_result=False)` / `.aio_run(..., wait_for_result=False)`, Small rewrite
[5](#step-5-run-a-hatchet-worker), `celery -A app worker`, `hatchet worker dev` or worker script, Small rewrite
[6](#step-6-migrate-retries-and-timeouts), `autoretry_for` / `self.retry()`, `retries` + `backoff_factor` + `NonRetryableException`, Small rewrite
[6](#step-6-migrate-retries-and-timeouts), `time_limit` / `soft_time_limit`, `execution_timeout` / `schedule_timeout`, Direct API swap
[7](#step-7-migrate-delayed-and-periodic-tasks), `apply_async(countdown=...)` / `eta=...`, `task.aio_schedule(run_at, input)`, Small rewrite
[7](#step-7-migrate-delayed-and-periodic-tasks), `beat_schedule` + `celery beat` process, `on_crons=["..."]` in task definition, Small rewrite
[8](#step-8-migrate-chains-groups-and-chords), `chain(a.s(), b.s())`, DAG workflow with `parents=[a]`, Conceptual redesign
[8](#step-8-migrate-chains-groups-and-chords), `group(a.s(), b.s())`, Parallel DAG tasks (no `parents`), Conceptual redesign
[8](#step-8-migrate-chains-groups-and-chords), `chord(group, callback)`, DAG task with multiple `parents`, Conceptual redesign
[9](#step-9-replace-result-backend-and-flower-monitoring), Result backend + `AsyncResult` + Flower, Hatchet run history + dashboard, Operational change
[3](#step-3-convert-task-definitions), `task_serializer` (pickle/msgpack), JSON via Pydantic, Small rewrite
[10](#step-10-migrate-other-celery-project-surfaces), `task_routes` / queues / routing, Worker registration + [worker affinity](/v1/advanced-assignment/worker-affinity), Conceptual redesign
[10](#step-10-migrate-other-celery-project-surfaces), Celery signals (`task_prerun`, etc.), `on_failure_task` / `on_success_task` / `ctx.log()`, Conceptual redesign
[10](#step-10-migrate-other-celery-project-surfaces), `revoke` / task cancellation, [Cancellation API](/v1/error-handling/cancellation) + dashboard + `ctx.is_cancelled`, Small rewrite
[10](#step-10-migrate-other-celery-project-surfaces), Task priority, [Priority](/v1/priority) (1-3 levels), Operational change
[10](#step-10-migrate-other-celery-project-surfaces), `task_always_eager` / testing, `.mock_run()` / `.aio_mock_run()`, Small rewrite
## Step 1: Update dependencies and runtime configuration
### Dependencies
Install `hatchet-sdk` alongside Celery using your chosen package manager. For example:
```bash
pip install hatchet-sdk
```
Celery and Hatchet can run side-by-side during a migration. This lets you move Celery tasks to Hatchet one at a time and validate behavior incrementally. Since Celery and Hatchet do not share a workflow runtime, each migrated task or workflow must have a clear boundary. Once a task or workflow is moved to Hatchet, update the application code that enqueues it to call Hatchet instead of Celery.
### Configuration
Celery projects configure broker and result backend connections. Hatchet does not have equivalent broker or result-backend settings. Instead, configure the Hatchet SDK so it can connect to the Hatchet engine. For [Hatchet Cloud](https://cloud.onhatchet.run), configure the SDK with an API token:
```bash
export HATCHET_CLIENT_TOKEN="your-token-here"
```
For [self-hosted](/self-hosting) or local Hatchet deployments, you may need environment-specific client settings for the Hatchet engine endpoint and TLS configuration.
Celery's default loader reads settings from a `celeryconfig.py` module on the Python path. Framework integrations may load the same settings from another source; for example, Django projects commonly load Celery settings from Django settings with a `CELERY_` namespace. See the [Celery configuration reference](https://docs.celeryq.dev/en/stable/userguide/configuration.html) for the full list. If your project has an extensive Celery configuration, review each section of this guide to determine which settings need migration and which can be removed.
### Process and deployment cleanup
Celery projects often run multiple long-lived processes, including workers, a task scheduler named Celery Beat, and optionally Flower for runtime monitoring. Depending on your environment, these may be managed through init scripts, systemd services, supervisor, launchd, Docker Compose, Kubernetes manifests, a `Procfile`, or CI/deployment scripts. Celery's [daemonization docs](https://docs.celeryq.dev/en/stable/userguide/daemonizing.html) provide guidance on managing these processes and are a useful resource for understanding an existing deployment.
Replace Celery worker and Beat processes with Hatchet [worker scripts](#step-5-run-a-hatchet-worker). Hatchet cron runs are managed by the Hatchet engine, so there is no separate scheduler process to deploy. Hatchet's dashboard replaces the basic Flower deployment path for runtime visibility.
Broker and result backend services are a separate cleanup step. Do not remove Redis, RabbitMQ, SQS, SQL databases, Cassandra, or other infrastructure just because Celery is being removed. Remove or decommission those services only after confirming they were used exclusively for Celery and are not still used by your application.
## Step 2: Replace the Celery app with a Hatchet client
A Celery project usually has a shared `Celery` app instance that defines the broker, result backend, and task registry. In Hatchet, replace that shared app object with a shared `Hatchet` client.
**Celery:**
```python
from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379", backend="redis://localhost:6379")
```
**Hatchet:**
```python
from hatchet_sdk import Hatchet
hatchet = Hatchet()
```
The `Hatchet()` client reads its SDK configuration from `HATCHET_CLIENT_*` environment variables by default, including `HATCHET_CLIENT_TOKEN`. Create a `Hatchet` instance in a shared module and import it wherever your code needs to interact with Hatchet.
For self-hosted or local deployments, you may need additional client configuration such as the Hatchet API endpoint, gRPC host, or TLS settings. These can be supplied through environment variables, and the SDK also supports explicit [client configuration](/reference/python/client) when needed.
What changed:
- **The shared app object changes.** Replace the shared Celery `app = Celery(...)` instance with a shared `hatchet = Hatchet()` client.
- **Broker and backend settings move out of the app constructor.** Hatchet does not take Celery-style `broker=` or `backend=` arguments.
- **Connection settings move to SDK configuration.** Use `HATCHET_CLIENT_*` environment variables, or explicit client configuration when needed.
- **The same client is reused across the migration.** Import the shared `hatchet` client where you define tasks, workflows, workers, and code that triggers runs.
## Step 3: Convert task definitions
Celery tasks accept optional task-specific arguments. In Hatchet, [task functions](/v1/tasks) receive one input object and a context object instead of positional args.
**Celery:**
```python
from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379")
@app.task
def process_image(image_url: str, filters: list[str]) -> dict:
result = resize(image_url, filters)
return {"processed_url": result}
```
To migrate the Celery task, define an input model for `image_url` and `filters`, change the decorator to `@hatchet.task(...)`, add `ctx: Context`, and return a serializable task output.
**Hatchet:**
```python
@hatchet.task(name="process-image", input_validator=ImageInput)
async def process_image(input: ImageInput, ctx: Context) -> ImageOutput:
result = await resize(input.image_url, input.filters)
return ImageOutput(processed_url=result)
```
What changed:
- **`@app.task` becomes `@hatchet.task(...)`.** Use `input_validator=YourModel` to validate and type the task input.
- **Positional arguments move into an input model.** Replace `(image_url, filters)` with `input: ImageInput`, where `ImageInput` is a Pydantic `BaseModel` with those fields.
- **The task receives a context object.** Add `ctx: Context` as the second argument for run metadata, retry information, parent task outputs, and logging.
- **The return value becomes the task output.** Return a value Pydantic can serialize, such as a Pydantic model or a dict.
- **Tasks can be sync or async.** Hatchet tasks can be `def` or `async def`; the SDK [recommends async](/reference/python/asyncio) for I/O-bound work.
> **Warning:** If your Celery tasks use positional arguments, `*args`, or `**kwargs`, you
> will need to define a Pydantic model that captures the expected fields. For
> tasks that take no meaningful input, use `EmptyModel` from `hatchet_sdk`. This
> input-model conversion is the main mechanical cost of the migration.
### Serialization
Celery supports serializers such as JSON, pickle, YAML, and msgpack through `task_serializer`. Hatchet [task inputs and outputs](/v1/tasks#input-and-output) are serialized through Pydantic. If your Celery project uses `pickle` or another non-JSON serializer, make sure your task payloads can be represented in Pydantic's [JSON serialization mode](https://pydantic.dev/docs/validation/latest/concepts/serialization/#json-mode). Values that Pydantic cannot serialize directly should be converted to JSON-compatible fields or handled with custom Pydantic serializers.
## Step 4: Invoke tasks with input models
In Celery, a function decorated with `@app.task` is invoked through Celery's task calling API. Calls such as `.delay(...)` pass arguments to the underlying task function, while `apply_async(...)` passes them through `args=` and `kwargs=`. In Hatchet, invoke the task with the input model you defined in Step 3. The same task input model is used whether you wait for the result or enqueue the task and continue.
**Celery:**
```python
# Fire-and-forget
process_image.delay("https://example.com/photo.png", ["thumbnail"])
# With options
process_image.apply_async(
args=["https://example.com/photo.png", ["thumbnail"]],
)
```
To migrate these calls, replace the task arguments with `ImageInput(...)` and call `.run()` or `.aio_run()` on the Hatchet task. By default, Hatchet waits for the task to finish and returns the typed result directly.
**Hatchet:**
```python
async def run_image_task() -> None:
# Wait for the result (default behavior)
result = await process_image.aio_run(
ImageInput(image_url="https://example.com/photo.png", filters=["thumbnail"]),
)
print(result.processed_url)
# Fire-and-forget: enqueue without waiting
ref = await process_image.aio_run(
ImageInput(image_url="https://example.com/photo.png", filters=["thumbnail"]),
wait_for_result=False,
)
print(ref.workflow_run_id) # available immediately
# await ref.aio_result() to retrieve the result later
```
What changed:
- **Task arguments move into the input model.** Replace task arguments such as `("https://example.com/photo.png", ["thumbnail"])` with `ImageInput(image_url=..., filters=...)`.
- **Invocation methods change.** Replace `.delay()` / `.apply_async()` with `.run()` for sync code or `.aio_run()` for async code.
- **The default call waits for the result.** Celery's `.delay()` enqueues work and returns an `AsyncResult` immediately. Hatchet's `.run()` and `.aio_run()` wait until the task completes and return the typed result directly.
- **Fire-and-forget uses `wait_for_result=False`.** To enqueue without waiting, pass `wait_for_result=False`. This returns a `TaskRunRef` with the run ID and `.result()` / `.aio_result()` methods you can call later.
- **Delayed execution uses a separate API.** Replace Celery `countdown` or `eta` with scheduled runs, covered in [Step 7](#step-7-migrate-delayed-and-periodic-tasks).
## Step 5: Run a Hatchet worker
Celery workers are typically started from the CLI or through a process manager:
**Celery:**
```bash
celery -A tasks worker --loglevel=info --concurrency=4
```
In Hatchet, the migration artifact is a Python worker script that explicitly registers the tasks and workflows it can execute:
**Hatchet:**
```python
def start_worker() -> None:
worker = hatchet.worker("image-worker", slots=4, workflows=[process_image])
worker.start()
```
During development, start the worker with the [Hatchet CLI](/reference/cli/running-workers-locally), which handles authentication and hot reloads on code changes:
```bash
hatchet worker dev
```
In production, run the worker script directly with your `HATCHET_CLIENT_TOKEN` set:
```bash
python worker.py
```
What changed:
- **The worker is defined in code.** Replace Celery's app-based task discovery with a worker script that registers executable tasks and workflows in the worker's `workflows=[...]` list.
- **Worker capacity moves to `slots`.** Replace Celery's `--concurrency=4` flag with the `slots` parameter on the Hatchet worker.
- **Startup differs by environment.** Use `hatchet worker dev` during local development, then run the worker script through your process manager or container runtime in production.
- **Celery pool settings do not migrate directly.** Celery supports pool types such as prefork, eventlet, gevent, and threads. Hatchet Python workers use the SDK's sync/async execution model with worker slots, so CPU-bound work may need threads, subprocesses, or separate workers.
## Step 6: Migrate retries and timeouts
### Retries
Celery supports automatic retries for specified exception types and manual retries inside the task body.
**Celery:**
```python
# Automatic: retry only on RequestError, with exponential backoff
@app.task(bind=True, autoretry_for=(RequestError,),
retry_backoff=True, max_retries=5, retry_backoff_max=60)
def call_api(self, order_id: str) -> dict:
return external_api_call(order_id)
# Manual: explicit retry with custom logic
@app.task(bind=True, max_retries=3)
def call_api_manual(self, order_id: str) -> dict:
try:
return external_api_call(order_id)
except RequestError as exc:
raise self.retry(exc=exc, countdown=30)
```
To migrate automatic retries, move the retry policy onto the Hatchet task decorator. Hatchet retries task failures when retries are configured, so model retryable failures by raising normal exceptions.
**Hatchet:**
```python
@hatchet.task(
name="call-api",
retries=5,
backoff_factor=2.0,
backoff_max_seconds=60,
execution_timeout=timedelta(seconds=30),
input_validator=OrderInput,
)
async def call_api(input: OrderInput, ctx: Context) -> dict[str, str]:
result = await external_api_call(input.order_id)
return {"status": result}
```
To prevent retries for known permanent failures, raise `NonRetryableException` from the task body:
```python
from hatchet_sdk.exceptions import NonRetryableException
# Inside a task: skip retry for a permanent failure
if response.status_code == 400:
raise NonRetryableException("Bad request: do not retry")
```
See [retry policies](/v1/error-handling/retry-policies) for the full set of retry options and behavior.
What changed:
- **Retry policy moves onto the task decorator.** Replace Celery retry options such as `max_retries`, `retry_backoff`, and `retry_backoff_max` with Hatchet's `retries`, `backoff_factor`, and `backoff_max_seconds`.
- **Review retry exception rules.** Celery's `autoretry_for=(...)` lists the exceptions that should trigger retries. In Hatchet, task failures are retryable when `retries > 0`; raise `NonRetryableException` for failures that should not retry.
- **Manual `self.retry()` logic needs redesign.** Hatchet does not provide a direct `self.retry()` equivalent inside the task body; task-level retry behavior is managed by the Hatchet engine.
### Timeouts
**Celery:**
```python
@app.task(time_limit=30, soft_time_limit=25)
def long_task():
...
```
Celery defaults to no task time limit unless you configure one. Hatchet uses an [`execution_timeout`](/v1/error-handling/timeouts) for how long a task may run and `schedule_timeout` for how long a task may wait in the queue before being cancelled. For migration purposes, Hatchet's `execution_timeout` is the closest match for Celery's `time_limit`. Celery's `soft_time_limit` has no exact Hatchet equivalent. If your task depends on soft time limits for cleanup, move that cleanup into explicit task logic during migration.
**Hatchet:**
```python
from datetime import timedelta
@hatchet.task(execution_timeout=timedelta(seconds=30))
async def long_task(input, ctx):
...
```
What changed:
- **Hard task limits move to `execution_timeout`.** Replace Celery `time_limit=30` with `execution_timeout=timedelta(seconds=30)`.
- **Queue wait time is controlled separately.** Hatchet's `schedule_timeout` controls how long a task may wait in the queue before being cancelled.
- **Soft timeout cleanup must be rewritten.** Celery's `soft_time_limit` has no exact Hatchet equivalent, so cleanup that depends on soft-timeout exceptions should become explicit task logic.
- **Review timeout defaults.** Celery task time limits are not enabled unless configured; Hatchet has default timeout behavior, so check the timeout docs before relying on implicit behavior.
## Step 7: Migrate delayed and periodic tasks
### Delayed execution
In Celery, delayed execution is configured at the call site with `countdown` or `eta` on `apply_async`:
```python
# Run 5 minutes from now
process_image.apply_async(args=["https://example.com/photo.png", ["blur"]], countdown=300)
# Run at a specific time
from datetime import datetime, timedelta, timezone
process_image.apply_async(
args=["https://example.com/photo.png", ["blur"]],
eta=datetime.now(timezone.utc) + timedelta(hours=1),
)
```
> **Warning:** Celery's documentation
> [warns](https://docs.celeryq.dev/en/stable/userguide/calling.html#eta-and-countdown)
> that ETA/countdown tasks remain in worker memory until their scheduled
> execution time, and recommends using short delays rather than scheduling far
> into the future.
When migrating those call sites to Hatchet, delayed execution moves to the scheduled run API. Scheduled runs are persisted by the Hatchet engine instead of being held in worker memory, which makes longer delays feasible:
```python
async def schedule_for_later() -> None:
from datetime import datetime, timezone
run_at = datetime.now(tz=timezone.utc) + timedelta(hours=1)
await process_image.aio_schedule(
run_at,
ImageInput(image_url="https://example.com/photo.png", filters=["blur"]),
)
```
What changed:
- **`countdown` / `eta` becomes scheduled runs.** Replace `apply_async(..., countdown=...)` or `apply_async(..., eta=...)` with `task.aio_schedule(run_at, input)`.
- **The task input shape stays the same.** Scheduled runs use the same `ImageInput(...)` model as immediate `.aio_run(...)` calls.
- **The worker no longer holds the delay.** Hatchet persists the scheduled run in the engine until it is ready to execute.
### Periodic tasks
In Celery, recurring schedules are usually defined in `beat_schedule` and executed by a separate Celery Beat process:
```python
# celeryconfig.py
from celery.schedules import crontab
beat_schedule = {
"daily-report": {
"task": "tasks.generate_report",
"schedule": crontab(hour=9, minute=0),
},
}
```
In production, Beat may be run through your process supervisor, container runtime, or directly from the CLI:
```bash
celery -A tasks beat # must run exactly one instance
```
In Hatchet, [cron triggers](/v1/cron-runs) are declared on the task with the `on_crons` parameter, which accepts a list of cron expressions managed by the Hatchet engine. When migrating periodic tasks to Hatchet, convert Beat schedules that use Celery's `crontab(...)` helper into `on_crons` entries:
```python
@hatchet.task(name="DailyReport", on_crons=["0 9 * * *"])
async def generate_report(input: EmptyModel, ctx: Context) -> dict[str, str]:
await build_report()
return {"status": "sent"}
```
What changed:
- **Celery Beat is removed.** You do not run a separate `celery beat` process or ensure that only one Beat instance is active.
- **The schedule moves onto the task.** Replace the `beat_schedule` entry with `on_crons=["0 9 * * *"]`.
- **Celery `crontab(...)` becomes a cron expression.** Use standard 5-field or 6-field cron syntax.
- **Non-crontab schedules need review.** Interval, solar, or custom Beat schedules may not convert directly to a cron expression.
- **Runtime schedule management moves to the cron client.** If your application creates, lists, or deletes schedules dynamically, use the [`hatchet.cron` client](/reference/python/feature-clients/cron).
## Step 8: Migrate chains, groups, and chords
This is the biggest conceptual change in the migration. Celery Canvas builds tasks at the _call site_. In contrast, Hatchet DAG workflows define the dependency graph up front in the workflow definition. Understanding Hatchet's [workflow orchestration model](/cookbooks/durable-tasks-vs-dags) will make the rest of this section easier to follow.
### Celery chain to Hatchet DAG
**Celery:**
```python
from celery import chain
pipeline = chain(
validate.s(order_id),
charge.s(),
fulfill.s(),
notify.s(),
)
pipeline.apply_async()
```
In a Celery chain, each task's return value is passed as the first argument to the next task. To migrate a Celery chain, turn each task in the chain into a task in the same Hatchet workflow, then express the order of execution with parents. The first task has no parent, the second task depends on the first, and each later task depends on the task that came before it.
**Hatchet:**
```python
order_pipeline = hatchet.workflow(name="OrderPipeline", input_validator=OrderInput)
@order_pipeline.task(execution_timeout=timedelta(seconds=30))
async def validate(input: OrderInput, ctx: Context) -> OrderValidated:
ok = await check_inventory(input.order_id)
return OrderValidated(order_id=input.order_id, valid=ok)
@order_pipeline.task(parents=[validate])
async def charge(input: OrderInput, ctx: Context) -> ChargeResult:
parent = ctx.task_output(validate)
cid = await process_charge(parent.order_id)
return ChargeResult(order_id=input.order_id, charge_id=cid)
@order_pipeline.task(parents=[charge])
async def fulfill(input: OrderInput, ctx: Context) -> FulfillResult:
parent = ctx.task_output(charge)
tracking = await ship_order(parent.order_id)
return FulfillResult(order_id=input.order_id, tracking_number=tracking)
@order_pipeline.task(parents=[fulfill])
async def notify(input: OrderInput, ctx: Context) -> NotifyResult:
parent = ctx.task_output(fulfill)
await send_notification(parent.order_id)
return NotifyResult(order_id=input.order_id, notified=True)
```
What changed:
- **Dependencies are declared on the task**, not at the call site. `parents=[validate]` means "run after `validate` finishes."
- **Parent outputs are accessed explicitly** via `ctx.task_output(parent_task)`, not passed as positional arguments.
- **The workflow is triggered as a unit.** Replace `pipeline.apply_async()` with `await order_pipeline.aio_run(OrderInput(...))`. You do not chain individual task calls.
### Celery group to parallel DAG tasks
A Celery `group` runs multiple tasks in parallel. If the parallel tasks are known when you define the workflow, migrate them to Hatchet as DAG tasks with the same parent, or with no parents if they can start immediately.
**Celery:**
```python
from celery import group
checks = group(
check_inventory.s(order_id),
check_fraud.s(order_id),
)
checks.apply_async()
```
To migrate this pattern, define both tasks in the same Hatchet workflow without making one depend on the other. If the Celery code also waits for and combines the group results, that is an additional aggregation step, so add a downstream convergence task with the parallel tasks as parents.
**Hatchet:**
```python
order_checks = hatchet.workflow(name="OrderChecks", input_validator=OrderInput)
@order_checks.task()
async def check_inventory_task(input: OrderInput, ctx: Context) -> CheckResult:
ok = await check_inventory(input.order_id)
return CheckResult(passed=ok)
@order_checks.task()
async def check_fraud(input: OrderInput, ctx: Context) -> CheckResult:
ok = await run_fraud_check(input.order_id)
return CheckResult(passed=ok)
```
What changed:
- **Parallelism moves into the workflow definition.** Tasks with no dependency between them can run concurrently.
- **The call site starts the workflow, not a `group(...)`.** You trigger the workflow as a unit instead of constructing a Canvas group dynamically.
- **Aggregation may become a convergence task.** If the Celery code waits for group results and combines them, add a downstream task with the parallel tasks as parents.
- **Runtime fan-out is a different pattern.** If your Celery group is built from a runtime list, such as `group(process.s(item) for item in items)`, migrate that separately using [child spawning](/v1/child-spawning).
### Dynamic Celery groups to child spawning
Some Celery groups are built from a list that is only known at runtime:
**Celery:**
```python
from celery import group
items = get_items_for_order(order_id)
result = group(process_item.s(item_id) for item_id in items)()
```
This is different from a static group of known tasks. A Hatchet DAG is defined ahead of time, so it is not the right fit when the number of child tasks depends on runtime input. To migrate this pattern, use [child spawning](/v1/child-spawning). The parent task receives the list, spawns one child run per item using `aio_run_many`, and collects the results:
```python
@hatchet.task(name="process-item", input_validator=ItemInput)
async def process_item(input: ItemInput, ctx: Context) -> ItemResult:
result = await do_work(input.item_id)
return ItemResult(item_id=input.item_id, status=result)
@hatchet.task(name="fan-out-items", input_validator=OrderInput)
async def fan_out_items(input: OrderInput, ctx: Context) -> dict[str, list[ItemResult]]:
items = await get_items_for_order(input.order_id)
results = await process_item.aio_run_many(
[
process_item.create_bulk_run_item(input=ItemInput(item_id=item_id))
for item_id in items
],
)
return {"results": results}
```
What changed:
- **Build the fan-out inside a parent task.** Instead of constructing a `group(...)` at the call site, the parent task computes the runtime list and calls `aio_run_many`.
- **Each item becomes a child run.** The child task or workflow receives one item from the runtime list.
- **Collect results where you spawn the children.** Replace `GroupResult` handling with the results returned to the parent task by `aio_run_many`.
### Celery chord to DAG convergence
A Celery `chord` runs a group of tasks in parallel and then runs a callback after every task in the group completes. When the parallel tasks are known ahead of time, migrate this pattern to a Hatchet DAG convergence task with multiple parents.
**Celery:**
```python
from celery import chord
result = chord(
[fetch_a.s(order_id), fetch_b.s(order_id), fetch_c.s(order_id)]
)(aggregate.s())
```
The `aggregate` callback receives a list of results from the group. To migrate a static chord, make the aggregate task depend on each parallel task and read each parent output explicitly.
**Hatchet:**
```python
@workflow.task(parents=[fetch_a, fetch_b, fetch_c])
async def aggregate(input, ctx):
a = ctx.task_output(fetch_a)
b = ctx.task_output(fetch_b)
c = ctx.task_output(fetch_c)
...
```
What changed:
- **Fan-in is declared with multiple parents.** A task with `parents=[fetch_a, fetch_b, fetch_c]` runs only after all listed parents complete.
- **Parent outputs are accessed individually.** Instead of receiving a list of group results as a callback argument, the aggregate task calls `ctx.task_output(parent)` for each parent.
- **Runtime-sized chords need child spawning.** If your Celery chord is built from a runtime-sized group, such as `chord([fetch.s(url) for url in urls])(aggregate.s())`, migrate it with child spawning and explicit result collection rather than a static DAG convergence task.
## Step 9: Replace result backend and Flower monitoring
In Celery, result retrieval and task-state inspection depend on a configured result backend. If your application calls `AsyncResult.get()` or checks `AsyncResult.state`, migrate those patterns to Hatchet's result-returning invocation methods, `TaskRunRef`, or the [runs client](/reference/python/feature-clients/runs).
Flower is a separate cleanup step. Since Hatchet records run status, logs, retry attempts, timing, and workflow relationships in the dashboard, Flower is not needed for basic runtime visibility after migration is complete.
Consider the following Celery snippet that enqueues a task, retrieves its result, stores a task ID for later lookup, and checks task state:
**Celery:**
```python
# Wait for the result
async_result = process_image.delay(
"https://example.com/photo.png",
["thumbnail"],
)
output = async_result.get(timeout=10)
print(output["processed_url"])
# Fire-and-forget, then retrieve later
async_result = process_image.apply_async(
args=["https://example.com/photo.png", ["blur"]],
)
task_id = async_result.id
# Check task state later
state = async_result.state
print(state) # PENDING, SUCCESS, FAILURE, RETRY, or REVOKED
# Retrieve the result when ready
output = async_result.get(timeout=10)
print(output["processed_url"])
```
> **Info:** If your application branches on Celery task states, review that logic during
> migration instead of renaming states one-for-one. For example, Celery only
> reports `STARTED` when `track_started=True` or `task_track_started` is
> configured. In Hatchet, logic that depends on `STARTED` usually maps to
> checking for a `RUNNING` run status.
When migrating result-handling code to Hatchet, first decide whether the caller needs the result immediately or only needs a run reference. Use `.run()` / `.aio_run()` when the caller should wait for the result. Use `wait_for_result=False` when the caller should enqueue the work and inspect or retrieve the run later.
**Hatchet:**
```python
async def result_handling_example() -> None:
# Wait for the result directly (replaces AsyncResult.get())
result = await process_image.aio_run(
ImageInput(image_url="https://example.com/photo.png", filters=["thumbnail"]),
)
print(result.processed_url)
# Fire-and-forget, then retrieve later (replaces AsyncResult pattern)
ref = await process_image.aio_run(
ImageInput(image_url="https://example.com/photo.png", filters=["blur"]),
wait_for_result=False,
)
run_id = ref.workflow_run_id # available immediately
# Check run status (replaces AsyncResult.state)
status = await hatchet.runs.aio_get_status(run_id)
print(status) # QUEUED, RUNNING, COMPLETED, FAILED, or CANCELLED
# Retrieve the result when ready
result = await ref.aio_result()
print(result.processed_url)
```
What changed:
- **Result retrieval is direct by default.** Replace `AsyncResult.get()` with `.run()` or `.aio_run()` when the caller should wait for the task result.
- **Fire-and-forget returns a run reference.** Use `wait_for_result=False` when the caller should enqueue work and continue. The returned `TaskRunRef` includes `.workflow_run_id` and `.result()` / `.aio_result()`.
- **State inspection moves to the runs client.** Replace `AsyncResult.state` checks with `hatchet.runs.aio_get_status(workflow_run_id)` when application code needs run status.
- **Flower becomes deployment cleanup.** Once the migrated workloads no longer run through Celery, remove the Flower process and use Hatchet's dashboard for run visibility.
## Step 10: Migrate other Celery project surfaces
This section covers Celery features that often appear in production projects but do not have a one-to-one Hatchet equivalent. For each one, identify the Celery behavior your application depends on, then migrate it to the closest Hatchet pattern.
### Queues and routing
Celery uses `task_routes` and named queues to control which workers handle which tasks:
**Celery:**
```python
# celeryconfig.py
task_routes = {
"tasks.process_image": {"queue": "image-processing"},
"tasks.send_email": {"queue": "notifications"},
}
```
```bash
celery -A tasks worker -Q image-processing
```
In Hatchet, routing is handled by registering specific tasks on each worker. To migrate your Celery routes, remove the queue route for that task and register the migrated Hatchet task on the worker that should run it.
**Hatchet:**
```python
def start_image_worker() -> None:
"""Register only image-processing tasks on this worker."""
worker = hatchet.worker(
"image-processing-worker",
slots=4,
workflows=[process_image],
)
worker.start()
```
What changed:
- **`task_routes` and queue-bound workers become worker registration.** Replace Celery queue routing with Hatchet workers that register the tasks and workflows they can execute in `workflows=[...]`.
- **Advanced assignment uses worker labels.** If you need weighted routing or capability-based assignment, Hatchet supports [worker affinity](/v1/advanced-assignment/worker-affinity) where tasks declare `desired_worker_labels` and workers advertise capabilities. This replaces exchange-based routing patterns.
- **This is a redesign, not a rename.** Simple queue-per-worker routing becomes explicit worker registration. Complex routing rules should be evaluated against worker affinity during migration.
### Signals, hooks, and progress updates
Celery provides [signals](https://docs.celeryq.dev/en/stable/userguide/signals.html) for task lifecycle events. A common pattern is logging or alerting on failure. Celery also supports `self.update_state(state="PROGRESS", meta={...})` for progress reporting within a running task.
**Celery:**
```python
from celery.signals import task_failure
@task_failure.connect
def handle_task_failure(sender=None, task_id=None, exception=None, **kwargs):
notify_ops_team(task_id, exception)
```
Migrate Celery signals and progress reporting to Hatchet workflow-level hooks and structured logging. If the signal handles workflow-level success or failure, migrate it to a workflow hook. If it reports progress, move that into the task with `ctx.log()` or streaming:
**Hatchet:**
```python
hook_example = hatchet.workflow(name="HookExample", input_validator=OrderInput)
@hook_example.task()
async def process_order(input: OrderInput, ctx: Context) -> dict[str, str]:
ctx.log(f"Processing order {input.order_id}")
await process_charge(input.order_id)
ctx.log(f"Order {input.order_id} charged")
return {"status": "charged"}
@hook_example.on_failure_task()
async def on_order_failure(input: OrderInput, ctx: Context) -> None:
ctx.log(f"Order {input.order_id} failed, notifying support")
@hook_example.on_success_task()
async def on_order_success(input: OrderInput, ctx: Context) -> None:
ctx.log(f"Order {input.order_id} completed successfully")
```
What changed:
- **`task_failure` / `task_success` signals become workflow hooks.** `@workflow.on_failure_task()` runs after any task in the workflow fails. `@workflow.on_success_task()` runs after all tasks succeed. These are tasks within the workflow, not global signal handlers.
- **Human-readable progress moves to logs.** Use [`ctx.log()`](/v1/logging) to send progress messages visible in the dashboard.
- **Live progress data moves to streams.** For real-time progress data that application code consumes, use [`ctx.put_stream()`](/v1/streaming) to push data to subscribers.
- **Per-task `task_prerun` / `task_postrun` signals have no direct equivalent.** If your project uses these for setup or teardown around individual tasks, consider [dependency injection](/reference/python/dependency-injection). Other cross-cutting behavior may require a small redesign.
### Cancellation
Celery cancels tasks with `revoke()` on an `AsyncResult`:
**Celery:**
```python
result = process_image.delay("https://example.com/photo.png", ["blur"])
result.revoke() # cancel if still pending
result.revoke(terminate=True) # also terminate if running
```
Hatchet supports [task cancellation](/v1/error-handling/cancellation) through the runs client, the dashboard, or concurrency strategies like `CANCEL_IN_PROGRESS`. To cancel a run programmatically, use `hatchet.runs.cancel(run_id)`. Running tasks should cooperate by checking `ctx.is_cancelled`:
**Hatchet:**
```python
ref = await process_image.aio_run(
ImageInput(image_url="https://example.com/photo.png", filters=["blur"]),
wait_for_result=False,
)
# Cancel a run by ID
await hatchet.runs.aio_cancel(ref.workflow_run_id)
# Inside a task: cooperate with cancellation
@hatchet.task()
async def long_running(input, ctx):
for batch in batches:
if ctx.is_cancelled:
return {"status": "cancelled"}
await process_batch(batch)
return {"status": "done"}
```
What changed:
- **`result.revoke()` becomes `hatchet.runs.cancel(...)` / `hatchet.runs.aio_cancel(...)`.** Celery cancels via the `AsyncResult` object. Hatchet cancels via the [runs client](/reference/python/feature-clients/runs) or the dashboard using the workflow run ID.
- **Running tasks must cooperate.** Celery's `revoke(terminate=True)` sends a signal to the worker process. Hatchet sets a cancellation flag that the task checks with `ctx.is_cancelled`. Long-running tasks should check the flag so they can stop promptly and run any cleanup logic.
- **Concurrency strategies can cancel automatically.** Hatchet's `CANCEL_IN_PROGRESS` strategy cancels existing runs when a new run arrives for the same concurrency key. This has no Celery equivalent.
### Priority
Celery priority is assigned when the task is enqueued, and the meaning of the numeric priority value depends on the broker.
**Celery:**
```python
process_image.apply_async(
args=["https://example.com/photo.png", ["blur"]],
priority=9, # broker-dependent scale
)
```
When migrating priority-sensitive task calls to Hatchet, replace broker-specific numeric priorities with Hatchet's priority enum. Hatchet supports [three priority levels](/v1/priority): `Priority.LOW`, `Priority.MEDIUM`, and `Priority.HIGH`. You can set priority when triggering a run, or define a default priority on the workflow.
**Hatchet:**
```python
from hatchet_sdk import Priority
ref = await process_image.aio_run(
ImageInput(image_url="https://example.com/photo.png", filters=["blur"]),
wait_for_result=False,
priority=Priority.HIGH,
)
```
What changed:
- **Broker-specific numbers become Hatchet priority levels.** Replace Celery numeric priorities with `Priority.LOW`, `Priority.MEDIUM`, or `Priority.HIGH`.
- **Priority can be set per run or as a workflow default.** Use per-run priority when priority changes by call site; use a workflow default when all runs of that workflow should share the same priority.
- **Priority is scoped to a workflow type.** Higher-priority runs of the same workflow are scheduled before lower-priority runs. Priority does not affect scheduling across different workflow types.
- **Priority-sensitive behavior needs validation.** Celery broker priority and Hatchet scheduling priority are different models, so validate workflows that depend on precise ordering.
### Testing migrated tasks
Celery projects sometimes use `task_always_eager` for testing, though Celery's own documentation [discourages it](https://docs.celeryq.dev/en/stable/userguide/testing.html) for unit tests:
**Celery:**
```python
task_always_eager = True
result = process_image.delay("https://example.com/photo.png", ["thumbnail"])
assert result.get()["processed_url"] == "https://cdn.example.com/photo.png"
```
Hatchet tasks can be unit-tested without a running engine using `.mock_run()` (sync) or `.aio_mock_run()` (async), which execute the task function directly with a mock context:
**Hatchet:**
```python
async def test_process_image() -> None:
result = await process_image.aio_mock_run(
input=ImageInput(
image_url="https://example.com/photo.png",
filters=["thumbnail"],
),
)
assert result.processed_url == "https://cdn.example.com/photo.png"
```
What changed:
- **`task_always_eager` is replaced by `.mock_run()` / `.aio_mock_run()`.** These execute the task function locally without connecting to the Hatchet engine, similar to calling the function directly but with a mock context that provides `retry_count`, `additional_metadata`, and `lifespan`.
- **Integration tests use `.run()` / `.aio_run()`.** To test against the full Hatchet engine, call `.run()` or `.aio_run()` with a running local or test instance.
- **See the [unit testing example](https://github.com/hatchet-dev/hatchet/tree/main/sdks/python/examples/unit_testing)** for patterns including sync, async, durable, and workflow-level mock testing.
## Migration caveats to review
Before finishing the migration, review these areas that require design decisions rather than mechanical code changes:
- **Task function signatures.** Every Hatchet task receives `(input, ctx)` instead of positional args. Creating Pydantic input models for tasks with varied argument signatures is typically the largest mechanical effort.
- **Canvas -> DAGs and child spawning.** Celery Canvas composition must be restructured into Hatchet DAG workflows or child spawning. This is a design change, not a rename.
- **Manual retries.** Celery's `self.retry()` has no Hatchet equivalent. Restructure around `NonRetryableException` or handle retries within the task for specific calls.
- **Lifecycle hooks.** Celery signals like `task_prerun` and `task_postrun` require workflow-level hooks, dependency injection, or restructuring in Hatchet.
- **Queue routing.** Celery's `task_routes` and named queues become explicit worker registration and optionally worker affinity (a different routing model).
## Hatchet-native features to adopt after migration
Once your tasks are running on Hatchet, these features go beyond Celery parity:
- **[Global rate limits](/v1/rate-limits):** Key-based rate limiting enforced across all workers. Celery's `rate_limit` is per-worker only.
- **[Concurrency strategies](/v1/concurrency):** Per-key concurrency control with strategies such as cancel-in-progress or cancel-newest. Celery has no per-task concurrency control.
- **[Durable sleep](/v1/durable-sleep):** Pause a workflow for minutes, hours, or days without holding a worker slot.
- **[Durable event waits](/v1/durable-event-waits):** Pause until an external event arrives. Useful for webhook-driven or human-in-the-loop workflows.
- **[Durable tasks](/v1/durable-tasks):** Imperative workflow composition with checkpointing for long-running, stateful workflows.
None of these are needed for a basic migration. They become useful when your workloads grow from background tasks into durable workflows.
After the basic migration is working, use [async APIs](/reference/python/asyncio) for I/O-bound tasks, [lifespans](/reference/python/lifespans) to initialize shared resources once per worker, and [dependency injection](/reference/python/dependency-injection) to pass common dependencies into tasks without wiring them manually in every function.
## Before migrating every workload
Migration does not have to be all-or-nothing. Consider keeping specific workloads on Celery if:
- They are simple fire-and-forget tasks that are already reliable and do not need orchestration or observability.
- They depend on Celery ecosystem integrations (django-celery-beat, django-celery-results, or broker-specific features) that would be costly to replace.
- They require extremely high throughput and do not need durable retention, observability, or workflow orchestration. Review Hatchet's [architecture and guarantees](/v1/architecture-and-guarantees#good-fit-for) before migrating those workloads.
## Final cleanup
After the last Celery task is migrated, remove the `celery` package declaration, including any Celery extras, from your dependency file and regenerate your lock file. Transitive dependencies like `kombu` and `amqp` will be removed automatically. Remove broker client packages (such as `redis`) only if no other code uses them. Delete `celeryconfig.py` and any Celery-specific environment variables (`broker_url`, `result_backend`, etc.) once nothing references them. Remove Celery worker, Beat, and Flower process definitions from your deployment configuration.
## Next steps
- [Quickstart](/v1/quickstart): set up Hatchet and run your first task
- [Tasks](/v1/tasks): task definition and configuration
- [Workers](/v1/workers): worker options
- [Retry policies](/v1/error-handling/retry-policies): retries and backoff
- [Timeouts](/v1/error-handling/timeouts): execution and scheduling timeouts
- [Scheduled runs](/v1/scheduled-runs): delayed execution
- [Cron runs](/v1/cron-runs): recurring schedules
- [DAG workflows](/v1/directed-acyclic-graphs): multi-step pipelines
- [Python SDK reference](/reference/python/client): full API reference
---
# Frequently Asked Questions
This page provides answers to a number of the most common questions we're asked, to help you keep making great use of Hatchet!
## How do I choose how many slots to set on my worker?
The default slot count for workers in Hatchet is 100. In many cases, leaving the default as-is will be perfectly fine, especially when first getting set up with Hatchet.
Over time, you'll likely run into one of two issues: Resource starvation (meaning the worker is using up too much memory, CPU, etc.), or wanting to squeeze more juice out of your workers.
If your workers are resource starved, there are basically two options:
1. Reduce the slot count, so the worker runs less work concurrently. This is a blunt instrument, in the sense that it doesn't let you _tune_ resources to the needs of the workload running on the worker. For instance, if you're using 100% of your memory but only 10% of your CPU, reducing the slot count will likely help the worker stay online, but you'll be significantly under-utilizing CPU. In this case, you can:
2. Reconfigure the specs of the machine the worker is running on. For instance, in the example above, you might be able to migrate from a CPU-optimized machine to a memory-optimized one, which will give you more efficient resource utilization across the board.
If only some of your tasks are heavy, [task slot cost](/v1/advanced-assignment/slot-cost) lets those tasks consume more than one slot, so the slot count can stay tuned for the light ones.
On the other hand, if your workers are underutilizing resources, your options are:
1. Increase the number of slots on them so they can pick up more work. This is especially helpful for heavily I/O bound tasks, which generally are spending most of their time waiting.
2. Similar to the opposite case of resource starvation, you can scale down the resource requirements of the machine the worker is running on.
> **Info:** In general, we recommend not pushing the number of slots on a single worker
> much past 250-300. At this point, it likely makes sense to scale more
> horizontally.
## Why am I seeing missed heartbeats and task reassignments?
Hatchet uses heartbeats to monitor worker health. Workers send a heartbeat every **4 seconds**. If the engine does not receive a heartbeat for **30 seconds**, the engine considers the worker to be inactive, and re-queues its in-flight tasks for other workers to pick up.
There are a number of common reasons a worker might miss heartbeats:
- **Process crash** - the worker process exits unexpectedly (OOM kill, unhandled exception, SIGKILL).
- **Network disruption** - the connection between the worker and the Hatchet engine is interrupted (DNS failure, firewall change, cloud network blip).
- **Resource pressure** - High CPU or memory usage can starve the worker for resources
---
# Cookbooks
Our cookbooks are guides to help you solve common problems you'll find easy to tackle with Hatchet.
## Webhooks
Receive webhooks from external services and use them to trigger tasks in Hatchet. Each guide walks through setup end-to-end — creating the webhook in Hatchet, wiring it up to the source, and writing a task that handles the incoming event.
Handle payment events, subscription changes, and other Stripe webhooks.
React to pushes, pull requests, issues, and other GitHub events.
Respond to Slack events like messages, reactions, and slash commands.
## Workflow Patterns
Build practical end-to-end workflows with Hatchet’s durable execution, event handling, and task orchestration primitives.
Build a durable support workflow that triages tickets, waits for customer
replies, and escalates on timeout.
Send a welcome email after signup, wait for onboarding completion, and send
a follow-up only if the user does not finish in time.
Build a DAG workflow that extracts text from a PDF, classifies it, and
summarizes the content.
Build a highly parallelized object processor for Amazon S3.
## Agent Patterns
Expose Hatchet workflows and tasks as tools for AI agents via MCP and agent SDKs.
Expose Hatchet workflows and tasks as MCP-compatible agent tools with
durable execution, retries, and observability behind every tool call.
Use the Claude Agent SDK with Hatchet-backed MCP tools in a trusted
environment.
Use the OpenAI Agents SDK with Hatchet-backed tools in a trusted
environment.
---
# Stripe Webhooks
Stripe sends webhooks for all sorts of events, such as payments succeeding, subscription cancellations, invoice creation, and so on. This guide walks through setting up webhooks from Stripe to trigger events directly in Hatchet.
## Setup
### Get your Stripe webhook signing secret
In the [Stripe Dashboard](https://dashboard.stripe.com/webhooks), you'll create a new webhook endpoint. Don't fill in the URL yet — you'll get that from Hatchet in the next step. See [Stripe's webhooks guide](https://docs.stripe.com/webhooks) for more details on setting this up.
For now, note the **signing secret** that Stripe generates for you (it starts with `whsec_`). You'll need this to tell Hatchet how to verify incoming requests.
### Create the webhook in Hatchet
In the Hatchet dashboard, go to **Webhooks** and create a new webhook with the following settings:
Field, Value
**Name**, `stripe` (or whatever you'd like)
**Source**, Stripe
**Event Key Expression**, `'stripe:' + input.type`
**Secret**, Your `whsec_...` signing secret
The event key expression here takes the `type` field from Stripe's payload (something like `payment_intent.created`) and prefixes it with `stripe:` so your event keys are namespaced. When a webhook from Stripe is ingested with an `input.type` of `payment_intent.created`, there will be a corresponding Hatchet event with a key of `stripe:payment_intent.created` created.
Once you've created the webhook, copy the URL that Hatchet generates.
### Add the URL to Stripe
Go back to the Stripe Dashboard webhook you created in step 1 and paste in the Hatchet webhook URL. Select the events you want to listen for (or just select all of them — Hatchet will only trigger workflows that match the event key).
### Write a task that listens for Stripe events
Now you just need a task with a matching `on_events` trigger. For example, to handle successful payments:
#### Python
```python
class StripeObject(BaseModel):
customer: str
amount: int
class StripeData(BaseModel):
object: StripeObject
class StripePaymentInput(BaseModel):
type: str
data: StripeData
class StripePaymentOutput(BaseModel):
customer: str
amount: int
@hatchet.task(
input_validator=StripePaymentInput,
on_events=["stripe:payment_intent.succeeded"],
)
def handle_stripe_payment(
input: StripePaymentInput, ctx: Context
) -> StripePaymentOutput:
customer = input.data.object.customer
amount = input.data.object.amount
print(f"Payment of {amount} from {customer}")
return StripePaymentOutput(customer=customer, amount=amount)
```
#### Typescript
```typescript
type StripePaymentInput = {
type: string;
data: {
object: {
customer: string;
amount: number;
};
};
};
export const handleStripePayment = hatchet.task({
name: 'handle-stripe-payment',
on: {
event: 'stripe:payment_intent.succeeded',
},
fn: async (input: StripePaymentInput, ctx) => {
const { customer, amount } = input.data.object;
ctx.logger.info(`Payment of ${amount} from ${customer}`);
return { customer, amount };
},
});
```
#### Go
```go
type StripePaymentInput struct {
Type string `json:"type"`
Data struct {
Object struct {
Customer string `json:"customer"`
Amount int `json:"amount"`
} `json:"object"`
} `json:"data"`
}
stripePayment := client.NewStandaloneTask(
"handle-stripe-payment",
func(ctx hatchet.Context, input StripePaymentInput) (*struct {
Customer string `json:"customer"`
Amount int `json:"amount"`
}, error) {
fmt.Printf("Payment of %d from %s\n", input.Data.Object.Amount, input.Data.Object.Customer)
return &struct {
Customer string `json:"customer"`
Amount int `json:"amount"`
}{
Customer: input.Data.Object.Customer,
Amount: input.Data.Object.Amount,
}, nil
},
hatchet.WithWorkflowEvents("stripe:payment_intent.succeeded"),
)
```
#### Ruby
```ruby
HANDLE_STRIPE_PAYMENT = HATCHET.task(
name: "handle-stripe-payment",
on_events: ["stripe:payment_intent.succeeded"]
) do |input, ctx|
customer = input["data"]["object"]["customer"]
amount = input["data"]["object"]["amount"]
puts "Payment of #{amount} from #{customer}"
{ "customer" => customer, "amount" => amount }
end
```
### Test it
You can use Stripe's "Send test webhook" feature in the dashboard, or trigger a real event in test mode. You should see the task run appear in the Hatchet dashboard.
---
# GitHub Webhooks
GitHub can send webhooks for repository events — pushes, pull requests, issues, releases, and so on. This guide walks through connecting GitHub webhooks to Hatchet.
## Setup
### Create the webhook in Hatchet
In the Hatchet dashboard, go to **Webhooks** and create a new webhook with the following settings:
Field, Value
**Name**, `github` (or whatever you'd like)
**Source**, GitHub
**Event Key Expression**, `'github:' + headers['x-github-event'] + ':' + input.action`
**Secret**, A secret string of your choosing (you'll use the same one in GitHub)
A quick note on the event key expression: GitHub sends the event type (like `pull_request` or `issues`) in the `x-github-event` header, and the specific action (like `opened` or `closed`) in the payload's `action` field. The expression above combines them to produce keys like `github:pull_request:opened`.
Not all GitHub events have an `action` field, though. Push events, for instance, don't. If you want to handle events that might not have an `action`, you could use a simpler expression like `'github:' + headers['x-github-event']` and handle action-level routing in your task logic instead. Or you could create two separate webhooks — one for action-based events and one for action-less events.
Once you've created the webhook, copy the URL.
### Configure the webhook in GitHub
Go to your repository (or organization) settings, find **Webhooks**, and add a new webhook. See [GitHub's webhook docs](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks) for the full walkthrough.
1. **Payload URL**: Paste the Hatchet webhook URL.
2. **Content type**: Select `application/json`.
3. **Secret**: Enter the same secret you used when creating the webhook in Hatchet.
4. **Events**: Choose "Let me select individual events" and pick the ones you care about, or select "Send me everything" if you prefer.
> **Warning:** Make sure you set the content type to `application/json`. GitHub defaults to
> `application/x-www-form-urlencoded`, which won't work with Hatchet's JSON
> payload parsing.
### Write a task that listens for GitHub events
Here's an example that triggers when a pull request is opened:
#### Python
```python
class GitHubPullRequest(BaseModel):
number: int
title: str
class GitHubRepository(BaseModel):
full_name: str
class GitHubPRInput(BaseModel):
action: str
pull_request: GitHubPullRequest
repository: GitHubRepository
class GitHubPROutput(BaseModel):
repo: str
pr: int
@hatchet.task(
input_validator=GitHubPRInput,
on_events=["github:pull_request:opened"],
)
def handle_github_pr(input: GitHubPRInput, ctx: Context) -> GitHubPROutput:
repo = input.repository.full_name
pr_number = input.pull_request.number
title = input.pull_request.title
print(f"PR #{pr_number} opened on {repo}: {title}")
return GitHubPROutput(repo=repo, pr=pr_number)
```
#### Typescript
```typescript
type GitHubPRInput = {
action: string;
pull_request: {
number: number;
title: string;
};
repository: {
full_name: string;
};
};
export const handleGitHubPR = hatchet.task({
name: 'handle-github-pr',
on: {
event: 'github:pull_request:opened',
},
fn: async (input: GitHubPRInput, ctx) => {
const repo = input.repository.full_name;
const prNumber = input.pull_request.number;
const { title } = input.pull_request;
ctx.logger.info(`PR #${prNumber} opened on ${repo}: ${title}`);
return { repo, pr: prNumber };
},
});
```
#### Go
```go
type GitHubPRInput struct {
Action string `json:"action"`
PullRequest struct {
Number int `json:"number"`
Title string `json:"title"`
} `json:"pull_request"`
Repository struct {
FullName string `json:"full_name"`
} `json:"repository"`
}
githubPR := client.NewStandaloneTask(
"handle-github-pr",
func(ctx hatchet.Context, input GitHubPRInput) (*struct {
Repo string `json:"repo"`
PR int `json:"pr"`
}, error) {
fmt.Printf("PR #%d opened on %s: %s\n", input.PullRequest.Number, input.Repository.FullName, input.PullRequest.Title)
return &struct {
Repo string `json:"repo"`
PR int `json:"pr"`
}{
Repo: input.Repository.FullName,
PR: input.PullRequest.Number,
}, nil
},
hatchet.WithWorkflowEvents("github:pull_request:opened"),
)
```
#### Ruby
```ruby
HANDLE_GITHUB_PR = HATCHET.task(
name: "handle-github-pr",
on_events: ["github:pull_request:opened"]
) do |input, ctx|
repo = input["repository"]["full_name"]
pr_number = input["pull_request"]["number"]
title = input["pull_request"]["title"]
puts "PR ##{pr_number} opened on #{repo}: #{title}"
{ "repo" => repo, "pr" => pr_number }
end
```
### Test it
After saving the webhook in GitHub, GitHub will send a `ping` event to verify the connection. You can also use the "Redeliver" button in GitHub's webhook settings to replay past events, or just open a PR to trigger a real event.
---
# Slack Webhooks
Slack has several different ways to send data to your app — slash commands, interactive components (buttons, modals, etc.), and event subscriptions. They each have different payload formats and authentication mechanisms, which makes the setup a bit more involved than other webhook integrations. This guide walks through each one.
> **Info:** Slack's different interaction modes use different content types. Event
> subscriptions send JSON, but slash commands and interactive components send
> form-encoded data. Hatchet handles both, but it's good to be aware of the
> difference when writing your CEL expressions and task logic.
## Slack App Setup
Before configuring anything in Hatchet, you'll need a Slack app. If you don't already have one:
1. Go to [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App**. See [Slack's getting started guide](https://api.slack.com/quickstart) if this is your first time.
2. Choose **From scratch**, give it a name, and select your workspace.
3. Once created, go to **Basic Information** and note the **Signing Secret** — you'll need this for Hatchet. See [Slack's signing secret docs](https://api.slack.com/authentication/verifying-requests-from-slack) for more on how request verification works.
## Event Subscriptions
Event subscriptions are what Slack uses to notify your app about things happening in the workspace — messages being posted, channels being created, users joining, and so on.
### Create the webhook in Hatchet
Field, Value
**Name**, `slack-events`
**Source**, Slack
**Event Key Expression**, `'slack:event:' + input.event.type`
**Secret**, Your Slack app's signing secret
Copy the generated URL.
### Enable Event Subscriptions in Slack
In your Slack app settings, go to [**Event Subscriptions**](https://api.slack.com/events), toggle it on, and paste the Hatchet webhook URL into the **Request URL** field.
> **Info:** Slack will send a challenge request to verify the URL. Hatchet handles this
> automatically — you should see a green checkmark confirming the URL is
> verified.
Then, under **Subscribe to bot events**, add the events you want to listen for (e.g., `message.channels`, `app_mention`, `member_joined_channel`).
### Write a task
#### Python
```python
class SlackEvent(BaseModel):
type: str
user: str
text: str
channel: str
class SlackEventInput(BaseModel):
event: SlackEvent
class SlackEventOutput(BaseModel):
handled: bool
@hatchet.task(
input_validator=SlackEventInput,
on_events=["slack:event:app_mention"],
)
def handle_slack_mention(input: SlackEventInput, ctx: Context) -> SlackEventOutput:
print(
f"Mentioned by {input.event.user} in {input.event.channel}: {input.event.text}"
)
return SlackEventOutput(handled=True)
```
#### Typescript
```typescript
type SlackEventInput = {
event: {
type: string;
user: string;
text: string;
channel: string;
};
};
export const handleSlackMention = hatchet.task({
name: 'handle-slack-mention',
on: {
event: 'slack:event:app_mention',
},
fn: async (input: SlackEventInput, ctx) => {
const { user, text, channel } = input.event;
ctx.logger.info(`Mentioned by ${user} in ${channel}: ${text}`);
return { handled: true };
},
});
```
#### Go
```go
type SlackEventInput struct {
Event struct {
Type string `json:"type"`
User string `json:"user"`
Text string `json:"text"`
Channel string `json:"channel"`
} `json:"event"`
}
slackMention := client.NewStandaloneTask(
"handle-slack-mention",
func(ctx hatchet.Context, input SlackEventInput) (*struct {
Handled bool `json:"handled"`
}, error) {
fmt.Printf("Mentioned by %s in %s: %s\n", input.Event.User, input.Event.Channel, input.Event.Text)
return &struct {
Handled bool `json:"handled"`
}{Handled: true}, nil
},
hatchet.WithWorkflowEvents("slack:event:app_mention"),
)
```
#### Ruby
```ruby
HANDLE_SLACK_MENTION = HATCHET.task(
name: "handle-slack-mention",
on_events: ["slack:event:app_mention"]
) do |input, ctx|
event = input["event"]
puts "Mentioned by #{event["user"]} in #{event["channel"]}: #{event["text"]}"
{ "handled" => true }
end
```
## Slash Commands
Slash commands work differently from event subscriptions. When a user types something like `/deploy production`, Slack sends a form-encoded POST to your configured URL. The payload includes the command, the text after it, the user, the channel, and a `response_url` you can use to send a response back.
### Create the webhook in Hatchet
Field, Value
**Name**, `slack-commands`
**Source**, Slack
**Event Key Expression**, `'slack:command:' + input.command`
**Secret**, Your Slack app's signing secret
Copy the generated URL.
> **Info:** Even though slash commands send form-encoded payloads, Hatchet parses them
> into a JSON object so you can use the same `input.field` syntax in your CEL
> expressions.
### Add the slash command in Slack
In your Slack app settings, go to [**Slash Commands**](https://api.slack.com/interactivity/slash-commands) and create a new command. Set the **Request URL** to the Hatchet webhook URL you just copied.
### Write a task
The `input.command` field includes the leading slash (e.g., `/deploy`), so your event key will look like `slack:command:/deploy`.
#### Python
```python
class SlackCommandInput(BaseModel):
command: str
text: str
user_name: str
response_url: str
class SlackCommandOutput(BaseModel):
command: str
args: str
@hatchet.task(
input_validator=SlackCommandInput,
on_events=["slack:command:/deploy"],
)
def handle_slack_command(input: SlackCommandInput, ctx: Context) -> SlackCommandOutput:
print(f"{input.user_name} ran {input.command} {input.text}")
return SlackCommandOutput(command=input.command, args=input.text)
```
#### Typescript
```typescript
type SlackCommandInput = {
command: string;
text: string;
user_name: string;
response_url: string;
};
export const handleSlackCommand = hatchet.task({
name: 'handle-slack-command',
on: {
event: 'slack:command:/deploy',
},
fn: async (input: SlackCommandInput, ctx) => {
ctx.logger.info(`${input.user_name} ran ${input.command} ${input.text}`);
return { command: input.command, args: input.text };
},
});
```
#### Go
```go
type SlackCommandInput struct {
Command string `json:"command"`
Text string `json:"text"`
UserName string `json:"user_name"`
ResponseURL string `json:"response_url"`
}
slackCommand := client.NewStandaloneTask(
"handle-slack-command",
func(ctx hatchet.Context, input SlackCommandInput) (*struct {
Command string `json:"command"`
Args string `json:"args"`
}, error) {
fmt.Printf("%s ran %s %s\n", input.UserName, input.Command, input.Text)
return &struct {
Command string `json:"command"`
Args string `json:"args"`
}{
Command: input.Command,
Args: input.Text,
}, nil
},
hatchet.WithWorkflowEvents("slack:command:/deploy"),
)
```
#### Ruby
```ruby
HANDLE_SLACK_COMMAND = HATCHET.task(
name: "handle-slack-command",
on_events: ["slack:command:/deploy"]
) do |input, ctx|
puts "#{input["user_name"]} ran #{input["command"]} #{input["text"]}"
{ "command" => input["command"], "args" => input["text"] }
end
```
## Interactive Components
Interactive components — buttons, menus, modals — send payloads to an **Interactivity Request URL** when a user interacts with them. These are also form-encoded, with the actual payload nested inside a `payload` field as a JSON string.
### Create the webhook in Hatchet
Field, Value
**Name**, `slack-interactions`
**Source**, Slack
**Event Key Expression**, `'slack:interaction:' + input.type`
**Secret**, Your Slack app's signing secret
### Enable Interactivity in Slack
In your Slack app settings, go to [**Interactivity & Shortcuts**](https://api.slack.com/interactivity/handling), toggle it on, and paste the Hatchet webhook URL into the **Request URL** field.
### Write a task
#### Python
```python
class SlackAction(BaseModel):
action_id: str
class SlackUser(BaseModel):
username: str
class SlackInteractionInput(BaseModel):
type: str
actions: list[SlackAction]
user: SlackUser
class SlackInteractionOutput(BaseModel):
action: str
@hatchet.task(
input_validator=SlackInteractionInput,
on_events=["slack:interaction:block_actions"],
)
def handle_slack_interaction(
input: SlackInteractionInput, ctx: Context
) -> SlackInteractionOutput:
action = input.actions[0]
print(f"{input.user.username} clicked button: {action.action_id}")
return SlackInteractionOutput(action=action.action_id)
```
#### Typescript
```typescript
type SlackInteractionInput = {
type: string;
actions: Array<{ action_id: string }>;
user: { username: string };
};
export const handleSlackInteraction = hatchet.task({
name: 'handle-slack-interaction',
on: {
event: 'slack:interaction:block_actions',
},
fn: async (input: SlackInteractionInput, ctx) => {
const [action] = input.actions;
ctx.logger.info(`${input.user.username} clicked button: ${action.action_id}`);
return { action: action.action_id };
},
});
```
#### Go
```go
type SlackInteractionInput struct {
Type string `json:"type"`
Actions []struct {
ActionID string `json:"action_id"`
} `json:"actions"`
User struct {
Username string `json:"username"`
} `json:"user"`
}
slackInteraction := client.NewStandaloneTask(
"handle-slack-interaction",
func(ctx hatchet.Context, input SlackInteractionInput) (*struct {
Action string `json:"action"`
}, error) {
action := input.Actions[0]
fmt.Printf("%s clicked button: %s\n", input.User.Username, action.ActionID)
return &struct {
Action string `json:"action"`
}{Action: action.ActionID}, nil
},
hatchet.WithWorkflowEvents("slack:interaction:block_actions"),
)
```
#### Ruby
```ruby
HANDLE_SLACK_INTERACTION = HATCHET.task(
name: "handle-slack-interaction",
on_events: ["slack:interaction:block_actions"]
) do |input, ctx|
action = input["actions"][0]
puts "#{input["user"]["username"]} clicked button: #{action["action_id"]}"
{ "action" => action["action_id"] }
end
```
---
# How to Create a Support Agent Using Hatchet
Many real-world workflows become difficult to manage once they involve multiple steps, long waits, human replies, and escalation rules. Support is one example, but the same pattern also shows up in onboarding, approvals, incident response, and other operational flows. In this cookbook, we will build a simple support agent that triages a ticket, generates an initial reply, and then waits for either a customer response or a timeout. If the customer replies, the workflow resolves. If no reply arrives in time, the workflow escalates the ticket to a human support agent.
## What this example builds
This example implements the following durable support workflow:
```mermaid
flowchart TD
A[Support ticket received] --> B[Triage the ticket]
B --> C[Generate initial reply]
C --> D[Wait for reply or timeout]
D --> E[Customer reply]
D --> F[Timeout fires]
E --> G[Resolve ticket]
F --> H[Escalate to human support]
```
Hatchet's durable execution model helps keep the whole interaction in one workflow rather than scattering it across separate queue jobs and ad hoc timers.
## Setup
### Prepare your environment
To run this example, you will need:
- a working local Hatchet environment or access to [Hatchet Cloud](https://cloud.hatchet.run)
- a Hatchet SDK example environment (see the [Quickstart](/v1/quickstart))
- optionally, an `ANTHROPIC_API_KEY` for live LLM replies
Without `ANTHROPIC_API_KEY`, the example runs using a fixed fallback reply. To use the live Claude path, you also need the Anthropic SDK installed for your language.
### Define the models
Start by defining the types for the workflow input and task outputs.
#### Python
```python
class SupportTicketInput(BaseModel):
ticket_id: str
customer_email: str
subject: str
body: str
class TriageOutput(BaseModel):
category: str
priority: str
class ReplyOutput(BaseModel):
message: str
class EscalationOutput(BaseModel):
reason: str
assigned_to: str
```
#### Typescript
```typescript
export type SupportTicketInput = {
ticketId: string;
customerEmail: string;
subject: string;
body: string;
};
export type TriageOutput = {
category: string;
priority: string;
};
export type ReplyOutput = {
message: string;
};
export type EscalationOutput = {
reason: string;
assignedTo: string;
};
```
The models keep the inputs and outputs for each task explicit, which makes the workflow easier to inspect and test.
### Add the workflow tasks
The durable workflow delegates its work to a few small [tasks](/v1/tasks).
First, add a task to classify the incoming ticket:
#### Python
```python
@hatchet.task(input_validator=SupportTicketInput)
async def triage_ticket(input: SupportTicketInput, ctx: Context) -> TriageOutput:
"""Classify the ticket into a category and priority."""
subject = input.subject.lower()
body = input.body.lower()
text = subject + " " + body
if any(word in text for word in ["bill", "charge", "payment", "invoice"]):
category = "billing"
elif any(word in text for word in ["login", "password", "auth", "access"]):
category = "account"
else:
category = "technical"
if any(word in text for word in ["urgent", "critical", "down", "outage"]):
priority = "high"
elif any(word in text for word in ["twice", "broken", "error"]):
priority = "medium"
else:
priority = "low"
return TriageOutput(category=category, priority=priority)
```
#### Typescript
```typescript
// Classify the ticket into a category and priority.
export const triageTicket = hatchet.task({
name: 'triage-ticket',
fn: async (input: SupportTicketInput) => {
const text = `${input.subject} ${input.body}`.toLowerCase();
let category: string;
if (['bill', 'charge', 'payment', 'invoice'].some((w) => text.includes(w))) {
category = 'billing';
} else if (['login', 'password', 'auth', 'access'].some((w) => text.includes(w))) {
category = 'account';
} else {
category = 'technical';
}
let priority: string;
if (['urgent', 'critical', 'down', 'outage'].some((w) => text.includes(w))) {
priority = 'high';
} else if (['twice', 'broken', 'error'].some((w) => text.includes(w))) {
priority = 'medium';
} else {
priority = 'low';
}
return { category, priority };
},
});
```
Next, add a task to generate the initial support reply. When `ANTHROPIC_API_KEY` is set, the task calls Claude to produce the reply. Otherwise it returns a fixed fallback response.
#### Python
```python
@hatchet.task(input_validator=SupportTicketInput)
async def generate_reply(input: SupportTicketInput, ctx: Context) -> ReplyOutput:
"""Generate an initial support reply using Claude."""
api_key = os.environ.get("ANTHROPIC_API_KEY")
if not api_key:
return ReplyOutput(
message=f"Thank you for contacting support about: {input.subject}. "
"We are looking into this and will get back to you shortly."
)
import importlib
anthropic = importlib.import_module("anthropic")
client = anthropic.AsyncAnthropic(api_key=api_key)
response = await client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=300,
messages=[
{
"role": "user",
"content": (
f"You are a friendly support agent. Write a brief, helpful initial "
f"reply to this support ticket.\n\n"
f"Subject: {input.subject}\n"
f"Message: {input.body}\n\n"
f"Keep the reply under 3 sentences."
),
}
],
)
text = response.content[0].text
return ReplyOutput(message=text)
```
#### Typescript
```typescript
// Generate an initial support reply using Claude.
export const generateReply = hatchet.task({
name: 'generate-reply',
fn: async (input: SupportTicketInput) => {
const apiKey = process.env.ANTHROPIC_API_KEY;
if (!apiKey) {
return {
message: `Thank you for contacting support about: ${input.subject}. We are looking into this and will get back to you shortly.`,
};
}
// eslint-disable-next-line @typescript-eslint/no-require-imports
const anthropic = require('@anthropic-ai/sdk');
const Anthropic = anthropic.default || anthropic;
const client = new Anthropic({ apiKey });
const response = await client.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 300,
messages: [
{
role: 'user' as const,
content:
`You are a friendly support agent. Write a brief, helpful initial ` +
`reply to this support ticket.\n\n` +
`Subject: ${input.subject}\n` +
`Message: ${input.body}\n\n` +
`Keep the reply under 3 sentences.`,
},
],
});
const [block] = response.content;
const text = block?.type === 'text' ? block.text : '';
return { message: text };
},
});
```
Finally, add a task to represent escalation to the support team:
#### Python
```python
@hatchet.task(input_validator=SupportTicketInput)
async def escalate_ticket(input: SupportTicketInput, ctx: Context) -> EscalationOutput:
"""Escalate an unresolved ticket to the human support team."""
return EscalationOutput(
reason=f"No customer reply within {TIMEOUT_SECONDS}s timeout",
assigned_to="support-team@example.com",
)
```
#### Typescript
```typescript
// Escalate an unresolved ticket to the human support team.
export const escalateTicket = hatchet.task({
name: 'escalate-ticket',
fn: async (input: SupportTicketInput) => {
return {
reason: `No customer reply within ${TIMEOUT_SECONDS}s timeout`,
assignedTo: 'support-team@example.com',
};
},
});
```
Keeping triage, reply generation, and escalation as separate tasks keeps the workflow itself small and makes each piece easier to reason about.
### Build the durable workflow
Now tie everything together in a [durable Hatchet workflow](/v1/durable-execution). A durable workflow is a good fit here because this interaction may stay open for some time while waiting for a customer reply. Hatchet persists the workflow state and its wait conditions, so the workflow can survive long delays, worker restarts, or even a worker crash, then continue later on another worker. That gives you a straightforward way to model the whole interaction without adding custom recovery logic.
#### Python
```python
@hatchet.durable_task(input_validator=SupportTicketInput)
async def support_agent(
input: SupportTicketInput, ctx: DurableContext
) -> dict[str, Any]:
# Step 1: Triage the ticket
triage = await triage_ticket.aio_run(input)
# Step 2: Generate an initial reply
reply = await generate_reply.aio_run(input)
# Step 3: Wait for a customer reply or timeout
now = await ctx.aio_now()
consider_events_since = now - timedelta(minutes=LOOKBACK_MINUTES)
wait_result = await ctx.aio_wait_for(
"await-customer-reply",
or_(
SleepCondition(timedelta(seconds=TIMEOUT_SECONDS)),
UserEventCondition(
event_key=REPLY_EVENT_KEY,
scope=input.ticket_id,
consider_events_since=consider_events_since,
),
),
)
# The or-group result is {"CREATE": {"": ...}}.
# Check whether the reply event condition was the one that resolved.
resolved_key = list(wait_result["CREATE"].keys())[0]
customer_replied = resolved_key == REPLY_EVENT_KEY
if not customer_replied:
# Step 4a: Timeout -> escalate
await escalate_ticket.aio_run(input)
return {
"ticket_id": input.ticket_id,
"status": "escalated",
"triage_category": triage.category,
"triage_priority": triage.priority,
"initial_reply": reply.message,
}
# Step 4b: Customer replied -> resolve
return {
"ticket_id": input.ticket_id,
"status": "resolved",
"triage_category": triage.category,
"triage_priority": triage.priority,
"initial_reply": reply.message,
}
```
#### Typescript
```typescript
export const supportAgent = hatchet.durableTask({
name: 'support-agent',
executionTimeout: '10m',
fn: async (input: SupportTicketInput, ctx) => {
// Step 1: Triage the ticket
const triage = await triageTicket.run(input);
// Step 2: Generate an initial reply
const reply = await generateReply.run(input);
// Step 3: Wait for a customer reply or timeout
const now = await ctx.now();
const considerEventsSince = new Date(
now.getTime() - durationToMs(LOOKBACK_WINDOW)
).toISOString();
const waitResult = await ctx.waitFor(
Or(
new SleepCondition(`${TIMEOUT_SECONDS}s`, TIMEOUT_LABEL),
new UserEventCondition(
REPLY_EVENT_KEY,
'',
REPLY_LABEL,
undefined,
input.ticketId,
considerEventsSince
)
)
);
// Determine which condition fired. ctx.waitFor returns
// { CREATE: {
