Timeouts in Hatchet
Timeouts are an important concept in Hatchet that allow you to control how long a task is allowed to run before it is considered to have failed. This is useful for ensuring that your tasks don’t run indefinitely and consume unnecessary resources. Timeouts in Hatchet are treated as failures and the task will be retried if specified.
There are two types of timeouts in Hatchet:
- Scheduling Timeouts (Default 5m) - the time a task is allowed to wait in the queue before it is cancelled
- Execution Timeouts (Default 60s) - the time a task is allowed to run before it is considered to have failed
Timeout Format
In Hatchet, timeouts are specified using a string in the format <number><unit>, where <number> is an integer and <unit> is one of:
sfor secondsmfor minuteshfor hours
For example:
10smeans 10 seconds4mmeans 4 minutes1hmeans 1 hour
If no unit is specified, seconds are assumed.
In the Python SDK, timeouts can also be specified as a datetime.timedelta
object.
Task-Level Timeouts
You can specify execution and scheduling timeouts for a task using the execution_timeout and schedule_timeout parameters when creating a task.
In these tasks, both timeouts are specified, meaning:
- If the task is not scheduled before the
schedule_timeoutis reached, it will be cancelled. - If the task does not complete before the
execution_timeoutis reached (after starting), it will be cancelled.
A timed out task does not guarantee that the task will be stopped immediately. The task will be stopped as soon as the worker is able to stop the task. See cancellation for more information.
Refreshing Timeouts
In some cases, you may need to extend the timeout for a task while it is running. This can be done by using the task context.
For example:
In this example, the task initially would exceed its execution timeout. But before it does, we call the refreshTimeout method, which extends the timeout and allows it to complete. Importantly, refreshing a timeout is an additive operation - the new timeout is added to the existing timeout. So for instance, if the task originally had a timeout of 30s and we call refreshTimeout("15s"), the new timeout will be 45s.
The task timeout can be refreshed multiple times within a task to further extend the timeout as needed.