OpenTelemetry

Hatchet supports exporting traces from your tasks to an OpenTelemetry Collector to improve visibility into your Hatchet tasks.

Usage

Setup

Hatchet’s SDK provides an instrumentor that auto-instruments Hatchet code if you opt in. Setup is straightforward:

First, install the otel extra with (e.g.) pip install hatchet-sdk[otel]. Then, import the instrumentor:

from path.to.your.trace.provider import trace_provider
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor
 
HatchetInstrumentor(tracer_provider=trace_provider).instrument()

You bring your own trace provider and plug it into the HatchetInstrumentor, call instrument, and that’s it!

Spans

By default, Hatchet creates spans at the following points in the lifecycle of a task run:

1. When a trigger is run on the client side, e.g. run() or push() is called.
2. When a worker handles a task event, such as starting to run the task or cancelling the task

In addition, you’ll get a handful of attributes set (prefixed by hatchet.) on the task run events, such as the task name and the worker ID, as well as success/failure states, and so on.

Some other important notes:

1. The instrumentor will automatically propagate the trace context between task runs, so if you spawn a task from another task, the child will correctly show up as a child of its parent in the trace waterfall.
2. You can exclude specific attributes from being attached to spans by providing the otel configuration option on the ClientConfig and passing a list of excluded_attributes, which come from this list.