OpenTelemetry

Hatchet supports exporting traces from your workflows to an OpenTelemetry Collector to improve visibility into your Hatchet tasks.

Usage

Setup

Hatchet’s SDK provides an instrumentor that auto-instruments Hatchet code, if you opt in. Setup is straightforward:

First, install the otel extra with (e.g.) pip install hatchet-sdk[otel]. Then, import the instrumentor:

from path.to.your.trace.provider import trace_provider
from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor
 
HatchetInstrumentor(tracer_provider=trace_provider).instrument()

You bring your own trace provider and plug it into the HatchetInstrumentor, call instrument, and that’s it!

Providing a traceparent

In some cases, you might also want to provide a traceparent so any spans created in Hatchet are children of a parent that was created elsewhere in your application. You can do that by providing a traceparent key in the additional_metadata field of corresponding options field the following methods:

• hatchet.event.push via the PushEventOptions
• hatchet.event.bulk_push via the BulkPushEventOptions
• Workflow.run via the TriggerWorkflowOptions (and similarly for all other flavors of run, like aio_run, run_nowait, etc.)
• Workflow.run_many via the TriggerWorkflowOptions (and similarly for all other flavors of run_many, like aio_run_many, etc.)

For example:

hatchet.event.push(
    "user:create",
    {'userId': '1234'},
    options=PushEventOptions(
      additional_metadata={
          "traceparent":"00-f1aff5c5ea45185eff2a06fd5c0ed6c5-6f4116aff54d54d1-01" ## example traceparent
      }
    )
)

The HatchetInstrumentor also has some methods for generating traceparents that might be helpful:

1. create_traceparent creates a traceparent
2. inject_traceparent_into_metadata injects a traceparent into the additional_metadata field

Spans

By default, Hatchet creates spans at the following points in the lifecycle of a workflow run:

1. When a trigger is run on the client side, e.g. run_workflow() or push() is called.
2. When a worker handles a task event, like starting running the task or cancelling the task

In addition, you’ll get a handful of attributes set (prefixed by hatchet.) on the task run events, such as the workflow name and the worker id, as well as success / failure states, and so on.