Internal OpenTelemetry traces

This page describes how to export traces emitted by Hatchet’s own services, including the engine and the API server, to an OpenTelemetry collector, for visibility into a self-hosted Hatchet instance. This is different from the OpenTelemetry page in the guide, which covers tracing your own application, task, and workflow code that runs on your workers. Internal trace export is off by default, and is enabled only when you set a collector URL.

Enable trace export

Set a collector URL on the Hatchet service you want to trace. Export uses OTLP over gRPC, so the URL is a host and port with no scheme:

SERVER_OTEL_COLLECTOR_URL=<otel-collector-host>:4317
SERVER_OTEL_SERVICE_NAME=hatchet-engine

By default the connection uses TLS. For a collector that does not terminate TLS, set SERVER_OTEL_INSECURE=true. If your collector requires authentication, set SERVER_OTEL_COLLECTOR_AUTH to the value Hatchet should send in the Authorization header. To export only a fraction of traces, set SERVER_OTEL_TRACE_ID_RATIO to a value between 0 and 1. This is OpenTelemetry head sampling, and is unrelated to Trace Sampling, which samples dashboard results stored in the database.

The engine and the API server are configured independently. Set these variables on each service you want to export from, and set a distinct SERVER_OTEL_SERVICE_NAME on each if you want to tell them apart in your collector.

Kubernetes and Helm

The Helm charts do not expose dedicated OpenTelemetry values. Set the SERVER_OTEL_* variables through the generic environment passthrough described in Configuring the Helm Chart. For the hatchet-stack and hatchet-ha charts this is sharedConfig.env:

sharedConfig:
  env:
    SERVER_OTEL_COLLECTOR_URL: "<otel-collector-host>:4317"

Values under sharedConfig.env apply to all backend services, so a SERVER_OTEL_SERVICE_NAME set here is shared by every service. To give the engine and API server distinct service names, set SERVER_OTEL_SERVICE_NAME on the individual service deployments instead.

What Hatchet emits

When export is enabled, the engine and the API server initialize tracing and can emit spans, including spans for database queries traced through otelpgx. Spans are namespaced under hatchet.run/. Each exporting service sets the service.name resource attribute, from SERVER_OTEL_SERVICE_NAME, along with library.language=go, and adds k8s.pod.name and k8s.namespace.name when K8S_POD_NAME and K8S_POD_NAMESPACE are set.

Configuration reference

These variables are also listed on the engine configuration options page, with the configuration file key shown in parentheses. A related setting, SERVER_OTEL_METRICS_ENABLED (otel.metricsEnabled), enables OpenTelemetry metrics export over the same collector URL.