Simple Task Retries

Hatchet provides a simple and effective way to handle failures in your tasks using a retry policy. This feature allows you to specify the number of times a task should be retried if it fails, helping to improve the reliability and resilience of your tasks.

How it works

When a task fails (i.e. throws an error or returns a non-zero exit code), Hatchet can automatically retry the task based on the retries configuration defined in the task object. Here’s how it works:

1. If a task fails and retries is set to a value greater than 0, Hatchet will catch the error and retry the task.
2. The task will be retried up to the specified number of times, with each retry being executed after a short delay to avoid overwhelming the system.
3. If the task succeeds during any of the retries, the task will continue as normal.
4. If the task continues to fail after exhausting all the specified retries, the task will be marked as failed.

This simple retry mechanism can help to mitigate transient failures, such as network issues or temporary unavailability of external services, without requiring complex error handling logic in your task code.

How to use task-level retries

To enable retries for a task, simply add the retries property to the task object in your task definition:

@simple_workflow.task(retries=3)
def always_fail(input: EmptyModel, ctx: Context) -> dict[str, str]:
    raise Exception("simple task failed")

export const retries = hatchet.task({
  name: 'retries',
  retries: 3,
  fn: async (_, ctx) => {
    throw new Error('intentional failure');
  },
});

retries := client.NewStandaloneTask("retries-task", func(ctx hatchet.Context, input RetriesInput) (*RetriesResult, error) {
	return nil, errors.New("intentional failure")
}, hatchet.WithRetries(3))

SIMPLE_RETRY_WORKFLOW.task(:always_fail, retries: 3) do |input, ctx|
  raise "simple task failed"
end

You can add the retries property to any task, and Hatchet will handle the retry logic automatically.

It’s important to note that task-level retries are not suitable for all types of failures. For example, if a task fails due to a programming error or an invalid configuration, retrying the task will likely not resolve the issue. In these cases, you should fix the underlying problem in your code or configuration rather than relying on retries.

Additionally, if a task interacts with external services or databases, you should ensure that the operation is idempotent (i.e. can be safely repeated without changing the result) before enabling retries. Otherwise, retrying the task could lead to unintended side effects or inconsistencies in your data.

Accessing the Retry Count in a Running Task

If you need to access the current retry count within a task, you can use the retryCount method available in the task context:

Exponential Backoff

Hatchet also supports exponential backoff for retries, which can be useful for handling failures in a more resilient manner. Exponential backoff increases the delay between retries exponentially, giving the failing service more time to recover before the next retry.

@backoff_workflow.task(
    retries=10,
    # 👀 Maximum number of seconds to wait between retries
    backoff_max_seconds=10,
    # 👀 Factor to increase the wait time between retries.
    # This sequence will be 2s, 4s, 8s, 10s, 10s, 10s... due to the maxSeconds limit
    backoff_factor=2.0,
)
def backoff_task(input: EmptyModel, ctx: Context) -> dict[str, str]:
    if ctx.retry_count < 3:
        raise Exception("backoff task failed")

    return {"status": "success"}

Bypassing Retry logic

The Hatchet SDKs each expose a NonRetryable exception, which allows you to bypass pre-configured retry logic for the task. If your task raises this exception, it will not be retried. This allows you to circumvent the default retry behavior in instances where you don’t want to or cannot safely retry. Some examples in which this might be useful include:

1. A task that calls an external API which returns a 4XX response code.
2. A task that contains a single non-idempotent operation that can fail but cannot safely be rerun on failure, such as a billing operation.
3. A failure that requires manual intervention to resolve.

@non_retryable_workflow.task(retries=1)
def should_not_retry(input: EmptyModel, ctx: Context) -> None:
    raise NonRetryableException("This task should not retry")

In these cases, even though retries is set to a non-zero number (meaning the task would ordinarily retry), Hatchet will not retry.

Python SDK Client Retry Behavior

The retry behavior described above is for task execution inside Hatchet. The Python SDK also has separate retry behavior for certain client-side REST and gRPC calls made by the SDK itself.

These client retries are configured separately from task retries and do not control whether a task is retried after failing in a worker.

Default client retry behavior

By default, the Python SDK retries certain client calls with exponential backoff, with max_attempts defaulting to 5.

REST API calls

gRPC calls

Configuring Python SDK client retries

The Python SDK exposes client retry configuration through TenacityConfig, either directly in ClientConfig or via environment variables.

import os
 
from hatchet_sdk import Hatchet
from hatchet_sdk.config import ClientConfig, HTTPMethod, TenacityConfig
 
hatchet = Hatchet(
    config=ClientConfig(
        token=os.environ["HATCHET_CLIENT_TOKEN"],
        tenacity=TenacityConfig(
            max_attempts=5,
            retry_429=False,
            retry_transport_errors=False,
            retry_transport_methods=[HTTPMethod.GET, HTTPMethod.DELETE],
        ),
    )
)

You can also configure these via environment variables:

Idempotency considerations

You can add non-idempotent methods to retry_transport_methods, but only do so if:

1. Your operations are idempotent (for example, because they use idempotency keys), or
2. You understand and accept the risk of duplicate operations

Retry timing

Python SDK client retries use exponential backoff with jitter. Fine-grained backoff timing is not currently configurable through TenacityConfig.

Conclusion

Hatchet’s task-level retry feature is a simple and effective way to handle transient failures in your tasks, improving the reliability and resilience of your tasks. By specifying the number of retries for each task, you can ensure that your tasks can recover from temporary issues without requiring complex error handling logic.

Remember to use retries judiciously and only for tasks that are idempotent and can safely be repeated. For more advanced retry strategies, such as exponential backoff or circuit breaking, stay tuned for future updates to Hatchet’s retry capabilities.