Add to Cursor Add to Claude Copy for LLM View as MD

OpenTelemetry

Hatchet supports exporting traces from your tasks to an OpenTelemetry Collector to improve visibility into your Hatchet tasks.

This page is about tracing your application, task, and workflow code. To export traces emitted by Hatchet’s own services on a self-hosted instance, see Internal OpenTelemetry traces.

Setup

Install the otel extra:

pip install hatchet-sdk[otel]

Then create the instrumentor and call instrument():

from hatchet_sdk.opentelemetry.instrumentor import HatchetInstrumentor
 
HatchetInstrumentor().instrument()

By default, HatchetInstrumentor creates a TracerProvider that sends spans to the Hatchet engine’s OTLP collector. You can also pass your own TracerProvider:

HatchetInstrumentor(
    tracer_provider=your_tracer_provider,
).instrument()

Options

Option	Type	Default	Description
`tracer_provider`	`TracerProvider`	—	Custom TracerProvider. If not set, one is created automatically.
`enable_hatchet_otel_collector`	`bool`	`True`	Send traces to the Hatchet engine’s OTLP collector.
`schedule_delay_millis`	`int`	—	Delay between consecutive exports of the BatchSpanProcessor.
`max_export_batch_size`	`int`	—	Maximum batch size for the BatchSpanProcessor.
`max_queue_size`	`int`	—	Maximum queue size for the BatchSpanProcessor.

Install the required OpenTelemetry packages:

npm install @opentelemetry/api @opentelemetry/instrumentation @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-grpc

const { registerInstrumentations } = require("@opentelemetry/instrumentation");
import { HatchetInstrumentor } from "@hatchet-dev/typescript-sdk/opentelemetry";
 
registerInstrumentations({
  instrumentations: [new HatchetInstrumentor()],
});

By default, HatchetInstrumentor sends spans to the Hatchet engine’s OTLP collector. You can disable this with enableHatchetCollector: false.

Options

Option	Type	Default	Description
`enableHatchetCollector`	`boolean`	`true`	Send traces to the Hatchet engine’s OTLP collector.
`clientConfig`	`object`	—	Override Hatchet client config for the collector connection.
`includeTaskNameInSpanName`	`boolean`	`false`	Append the task action ID to the `hatchet.start_step_run` span name.
`excludedAttributes`	`array`	`[]`	List of `hatchet.*` attribute keys to exclude from spans.

Import the opentelemetry package (commonly aliased as hatchetotel):

import hatchetotel "github.com/hatchet-dev/hatchet/sdks/go/opentelemetry"

Create the instrumentor, then register its middleware on the worker:

instrumentor, err := hatchetotel.NewInstrumentor()
if err != nil {
    log.Fatalf("failed to create instrumentor: %v", err)
}
 
worker.Use(instrumentor.Middleware())

By default, NewInstrumentor creates a TracerProvider that sends spans to the Hatchet engine’s OTLP collector.

Remember to shut down the instrumentor on exit to flush remaining spans:

defer instrumentor.Shutdown(context.Background())

Options

Option	Description
`hatchetotel.WithTracerProvider(tp)`	Use a custom `*sdktrace.TracerProvider` instead of creating a new one.
`hatchetotel.DisableHatchetCollector()`	Disable sending traces to the Hatchet engine’s OTLP collector.
`hatchetotel.WithBatchSpanProcessorOptions(...)`	Configure the `BatchSpanProcessor` for the Hatchet collector.

Spans

By default, Hatchet creates spans at the following points in the lifecycle of a task run:

Producer spans — when a trigger is called on the client side (e.g. run(), runNoWait(), push(), schedule()).
Consumer spans — when a worker handles a task run, such as starting to run the task (hatchet.start_step_run) or cancelling a task (hatchet.cancel_step_run).

Span Names

Span Name	Kind	Description
`hatchet.start_step_run`	`CONSUMER`	Worker begins executing a task.
`hatchet.cancel_step_run`	`CONSUMER`	Worker cancels a running task.
`hatchet.run_workflow`	`PRODUCER`	Client triggers a workflow run.
`hatchet.run_workflows`	`PRODUCER`	Client triggers multiple workflow runs.
`hatchet.schedule_workflow`	`PRODUCER`	Client creates a scheduled workflow run.
`hatchet.push_event`	`PRODUCER`	Client pushes an event.
`hatchet.bulk_push_event`	`PRODUCER`	Client pushes events in bulk.
`hatchet.durable.wait_for`	`INTERNAL`	Durable task waits for a signal/condition.

Span Attributes

All spans include an instrumentor attribute set to "hatchet". Consumer spans (hatchet.start_step_run) include the following hatchet.* attributes:

Attribute	Type	Description
`hatchet.tenant_id`	`string`	The tenant ID for the task run.
`hatchet.worker_id`	`string`	The worker handling the task.
`hatchet.workflow_run_id`	`string`	The workflow run ID.
`hatchet.step_run_id`	`string`	The task run ID.
`hatchet.step_id`	`string`	The task ID.
`hatchet.workflow_name`	`string`	The workflow name.
`hatchet.action_name`	`string`	The action ID (format: `workflowName:taskName`).
`hatchet.step_name`	`string`	The human-readable task name.
`hatchet.retry_count`	`int`	Current retry attempt (0-indexed).
`hatchet.workflow_id`	`string`	The workflow definition ID (if available).
`hatchet.workflow_version_id`	`string`	The workflow version ID (if available).
`hatchet.parent_workflow_run_id`	`string`	Parent workflow run ID (for child workflows).
`hatchet.child_workflow_index`	`int`	Child workflow index (for child workflows).
`hatchet.child_workflow_key`	`string`	Child workflow key (for child workflows).

Producer spans (hatchet.run_workflow) include hatchet.step_name (the workflow being triggered) and, on success, hatchet.child_workflow_run_id.

Context Propagation

All SDKs:

Automatically inject W3C traceparent into additionalMetadata on producer spans, so consumer spans on the worker become children of the trigger span.
Provide an HatchetAttributeSpanProcessor that propagates hatchet.* attributes from the parent task run span to all child spans created within the task. This means any custom spans you create inside a task function will automatically carry the same hatchet.* attributes.

Logging Worker Health Checks

We use cookies

OpenTelemetry

Setup

Options

Options

Options

Spans

Span Names

Span Attributes

Context Propagation