Simple Task Retries

Hatchet provides a simple and effective way to handle failures in your tasks using a retry policy. This feature allows you to specify the number of times a task should be retried if it fails, helping to improve the reliability and resilience of your tasks.

How it works

When a task fails (i.e. throws an error or returns a non-zero exit code), Hatchet can automatically retry the task based on the retries configuration defined in the task object. Here’s how it works:

1. If a task fails and retries is set to a value greater than 0, Hatchet will catch the error and retry the task.
2. The task will be retried up to the specified number of times, with each retry being executed after a short delay to avoid overwhelming the system.
3. If the task succeeds during any of the retries, the task will continue as normal.
4. If the task continues to fail after exhausting all the specified retries, the task will be marked as failed.

This simple retry mechanism can help to mitigate transient failures, such as network issues or temporary unavailability of external services, without requiring complex error handling logic in your task code.

How to use task-level retries

To enable retries for a task, simply add the retries property to the task object in your task definition:

@simple_workflow.task(retries=3)
def always_fail(input: EmptyModel, ctx: Context) -> dict[str, str]:
    raise Exception("simple task failed")

export const retries = hatchet.task({
  name: 'retries',
  retries: 3,
  fn: async (_, ctx) => {
    throw new Error('intentional failure');
  },
});

retries := client.NewStandaloneTask("retries-task", func(ctx hatchet.Context, input RetriesInput) (*RetriesResult, error) {
	return nil, errors.New("intentional failure")
}, hatchet.WithRetries(3))

SIMPLE_RETRY_WORKFLOW.task(:always_fail, retries: 3) do |input, ctx|
  raise "simple task failed"
end

You can add the retries property to any task, and Hatchet will handle the retry logic automatically.

It’s important to note that task-level retries are not suitable for all types of failures. For example, if a task fails due to a programming error or an invalid configuration, retrying the task will likely not resolve the issue. In these cases, you should fix the underlying problem in your code or configuration rather than relying on retries. See Bypassing retry logic.

Additionally, if a task interacts with external services or databases, you should ensure that the operation is idempotent (i.e. can be safely repeated without changing the result) before enabling retries. Otherwise, retrying the task could lead to unintended side effects or inconsistencies in your data.

Accessing the Retry Count in a Running Task

You can access the current retry count on the task’s context object:

@simple_workflow.task(retries=3)
def fail_twice(input: EmptyModel, ctx: Context) -> dict[str, str]:
    if ctx.retry_count < 2:
        raise Exception("simple task failed")

    return {"status": "success"}

export const retriesWithCount = hatchet.task({
  name: 'retries-with-count',
  retries: 3,
  fn: async (_, ctx) => {
    // > Get the current retry count
    const retryCount = ctx.retryCount();

    ctx.logger.info(`Retry count: ${retryCount}`);

    if (retryCount < 2) {
      throw new Error('intentional failure');
    }

    return {
      message: 'success',
    };
  },
});

retriesWithCount := client.NewStandaloneTask("fail-twice-task", func(ctx hatchet.Context, input RetriesWithCountInput) (*RetriesWithCountResult, error) {
	// Get the current retry count
	retryCount := ctx.RetryCount()

	fmt.Printf("Retry count: %d\n", retryCount)

	if retryCount < 2 {
		return nil, errors.New("intentional failure")
	}

	return &RetriesWithCountResult{
		Message: "success",
	}, nil
}, hatchet.WithRetries(3))

SIMPLE_RETRY_WORKFLOW.task(:fail_twice, retries: 3) do |input, ctx|
  raise "simple task failed" if ctx.retry_count < 2

  { "status" => "success" }
end

Exponential Backoff

Hatchet also supports exponential backoff for retries, which can be useful for handling failures in a more resilient manner. Exponential backoff increases the delay between retries exponentially, giving the failing service more time to recover before the next retry.

@backoff_workflow.task(
    retries=10,
    # 👀 Maximum number of seconds to wait between retries
    backoff_max_seconds=10,
    # 👀 Factor to increase the wait time between retries.
    # This sequence will be 2s, 4s, 8s, 10s, 10s, 10s... due to the maxSeconds limit
    backoff_factor=2.0,
)
def backoff_task(input: EmptyModel, ctx: Context) -> dict[str, str]:
    if ctx.retry_count < 3:
        raise Exception("backoff task failed")

    return {"status": "success"}

Bypassing Retry logic

The Hatchet SDKs each expose a NonRetryable exception, which allows you to bypass pre-configured retry logic for the task. If your task raises this exception, it will not be retried. This allows you to circumvent the default retry behavior in instances where you don’t want to or cannot safely retry. Some examples in which this might be useful include:

1. A task that calls an external API which returns a 4XX response code.
2. A task that contains a single non-idempotent operation that can fail but cannot safely be rerun on failure, such as a billing operation.
3. A failure that requires manual intervention to resolve.

@non_retryable_workflow.task(retries=1)
def should_not_retry(input: EmptyModel, ctx: Context) -> None:
    raise NonRetryableException("This task should not retry")

In these cases, even though retries is set to a non-zero number (meaning the task would ordinarily retry), Hatchet will not retry.

Python SDK Client Retry Behavior

The retry behavior described above is for task execution inside Hatchet. The Python SDK also has separate retry behavior for certain client-side REST and gRPC calls made by the SDK itself.

These client retries are configured separately from task retries and do not control whether a task is retried after failing in a worker.

Default client retry behavior

By default, the Python SDK retries certain client calls with exponential backoff, with max_attempts defaulting to 5.

REST API calls

gRPC calls

Configuring Python SDK client retries

The Python SDK exposes client retry configuration through TenacityConfig, either directly in ClientConfig or via environment variables.

import os
 
from hatchet_sdk import Hatchet
from hatchet_sdk.config import ClientConfig, HTTPMethod, TenacityConfig
 
hatchet = Hatchet(
    config=ClientConfig(
        token=os.environ["HATCHET_CLIENT_TOKEN"],
        tenacity=TenacityConfig(
            max_attempts=5,
            retry_429=False,
            retry_transport_errors=False,
            retry_transport_methods=[HTTPMethod.GET, HTTPMethod.DELETE],
        ),
    )
)

You can also configure these via environment variables:

Idempotency considerations

You can add non-idempotent methods to retry_transport_methods, but only do so if:

1. Your operations are idempotent (for example, because they use idempotency keys), or
2. You understand and accept the risk of duplicate operations

Retry timing

Python SDK client retries use exponential backoff with jitter. Fine-grained backoff timing is not currently configurable through TenacityConfig.

Go SDK Client Retry Behavior

The retry behavior described above is for task execution inside Hatchet. The Go SDK also retries some REST and gRPC calls that the SDK itself makes to Hatchet.

These SDK client retries are configured separately from task retries. They do not control whether Hatchet retries a task after it fails in a worker.

Default Go client retry behavior

By default, the Go SDK retries certain client calls with exponential backoff. REST reads use up to 5 total attempts: the initial attempt plus up to 4 retries. gRPC calls keep the existing 5 attempt retry limit.

REST read retries use bounded jittered backoff. When the caller request context has no deadline, each REST attempt uses a response-header timeout without cutting off response body reads. If the caller context already has a deadline, that deadline governs the whole request.

REST API calls (bodyless GET and HEAD only)

For HTTP 429 responses on idempotent reads, the Go SDK honors a valid Retry-After header when it fits the client retry cap. When Retry-After is missing, invalid, or oversized, it falls back to the same bounded jittered backoff used for other retriable errors.

gRPC calls

Configuring Go SDK client retries

Use environment variables to disable SDK client retries:

If both variables are set, all SDK client retries are disabled.

Idempotency considerations

Conclusion

Hatchet’s task-level retry feature is a simple and effective way to handle transient failures in your tasks, improving the reliability and resilience of your tasks. By specifying the number of retries for each task, you can ensure that your tasks can recover from temporary issues without requiring complex error handling logic.

Remember to use retries judiciously and only for tasks that are idempotent. For more advanced retry strategies, such as exponential backoff or circuit breaking, stay tuned for future updates to Hatchet’s retry capabilities.