We use cookies

We use cookies to ensure you get the best experience on our website. For more information on how we use cookies, please see our cookie policy.

By clicking "Accept", you agree to our use of cookies.
Learn more.

Self-HostingPrometheus Metrics
Add to CursorAdd to ClaudeCopy for LLMView as MD

Prometheus Metrics for Hatchet

This document provides an overview of the Prometheus metrics exposed by Hatchet, setup instructions for the metrics endpoint, and example PromQL queries to analyze them.

Setup

To enable Prometheus metrics for your Hatchet instance, you can set the following environment variables. The corresponding configuration YAML values are mentioned in parentheses. If you are deploying Hatchet in HA mode, these should be set on the grpc, controllers, and scheduler deployments.

⚠️

The global metrics are per-process counters — every engine pod maintains its own values, and different metrics are emitted by different roles. Task-creation counters in particular (hatchet_created_tasks_total and hatchet_tenant_created_tasks) increment on the grpc role, so a scrape that only covers controllers and scheduler will never see them, and example queries that reference created tasks (e.g. Queue Processing Efficiency: Assigned vs Created) cannot be computed. Make sure your scrape configuration targets all engine pods — including any pod running the combined all services — and aggregate the counters across pods (e.g. with sum(...)) when querying.

  • Required

    • SERVER_PROMETHEUS_ENABLED (prometheus.enabled)
      • Default: false
      • Description: Enables or disables the Prometheus metrics HTTP server.

  • Optional

    • SERVER_PROMETHEUS_ADDRESS (prometheus.address)

      • Default: ":9090"
      • Description: The network address and port to bind the Prometheus metrics server to.

    • SERVER_PROMETHEUS_PATH (prometheus.path)

      • Default: "/metrics"
      • Description: The HTTP path at which metrics will be exposed.

Once enabled, you can setup any scraper that supports ingesting Prometheus metrics.

Tenant metrics endpoint

This step requires communication with a service that scrapes Hatchet Prometheus metrics.

To enable the tenant API endpoint you can set the following environment variables:

  • Required

    • SERVER_PROMETHEUS_SERVER_URL (prometheus.prometheusServerURL)
      • Description: The Prometheus server URL.

  • Optional

    • SERVER_PROMETHEUS_SERVER_USERNAME (prometheus.prometheusServerUsername)

      • Description: The username to access the Prometheus instance via HTTP basic auth.

    • SERVER_PROMETHEUS_SERVER_PASSWORD (prometheus.prometheusServerPassword)

      • Description: The password to access the Prometheus instance via HTTP basic auth.

Example environment setup:

export SERVER_PROMETHEUS_ENABLED=true
export SERVER_PROMETHEUS_ADDRESS=":9999"
export SERVER_PROMETHEUS_PATH="/custom-metrics"

Restart your Hatchet server after setting these variables to apply the changes.

Global Metrics

Metric NameTypeDescription
hatchet_queue_invocations_totalCounterThe total number of invocations of the queuer function
hatchet_created_tasks_totalCounterThe total number of tasks created
hatchet_retried_tasks_totalCounterThe total number of tasks retried
hatchet_succeeded_tasks_totalCounterThe total number of tasks that succeeded
hatchet_failed_tasks_totalCounterThe total number of tasks that failed (in a final state, not including retries)
hatchet_skipped_tasks_totalCounterThe total number of tasks that were skipped
hatchet_cancelled_tasks_totalCounterThe total number of tasks cancelled
hatchet_assigned_tasksCounterThe total number of tasks assigned to a worker
hatchet_scheduling_timed_outCounterThe total number of tasks that timed out while waiting to be scheduled
hatchet_rate_limitedCounterThe total number of tasks that were rate limited
hatchet_queued_to_assignedCounterThe total number of unique tasks that were queued and later assigned to a worker
hatchet_queued_to_assigned_time_secondsHistogramBuckets of time (in seconds) spent in the queue before being assigned to a worker
hatchet_reassigned_tasksCounterThe total number of tasks that were reassigned to a worker

Example PromQL Queries

1. Rate of calls to the queuer method
rate(hatchet_queue_invocations_total[5m])
2. Average queue time in milliseconds
# Calculates average queue time over the past 5 minutes, converted to ms
rate(hatchet_queued_to_assigned_time_seconds_sum[5m])
  / rate(hatchet_queued_to_assigned_time_seconds_count[5m])
  * 1e3
3. Success and failure rates
rate(hatchet_succeeded_tasks_total[5m])
rate(hatchet_failed_tasks_total[5m])
4. Queue time distribution (histogram)
sum by (le) (
  rate(hatchet_queued_to_assigned_time_seconds_bucket[5m])
)
5. Rate of tasks created vs. retried
rate(hatchet_created_tasks_total[5m])
rate(hatchet_retried_tasks_total[5m])
6. Task Assignment Rate
rate(hatchet_assigned_tasks[5m])
7. Scheduling Timeout Rate
rate(hatchet_scheduling_timed_out[5m])
8. Rate Limiting Impact
rate(hatchet_rate_limited[5m])
9. Task Completion Ratio (Success vs Total)
rate(hatchet_succeeded_tasks_total[5m])
/
(rate(hatchet_succeeded_tasks_total[5m]) + rate(hatchet_failed_tasks_total[5m]))
10. Task Cancellation Rate
rate(hatchet_cancelled_tasks_total[5m])
11. Task Skip Rate
rate(hatchet_skipped_tasks_total[5m])
12. Queue Processing Efficiency (Assigned vs Created)
rate(hatchet_assigned_tasks[5m]) / rate(hatchet_created_tasks_total[5m])
13. Task Reassignment Rate
rate(hatchet_reassigned_tasks[5m])

Tenant Metrics

Metric NameTypeDescription
hatchet_tenant_workflow_duration_millisecondsHistogramDuration of workflow execution in milliseconds (DAGs and single tasks)
hatchet_tenant_queue_invocationsCounterThe total number of invocations of the queuer function
hatchet_tenant_created_tasksCounterThe total number of tasks created
hatchet_tenant_retried_tasksCounterThe total number of tasks retried
hatchet_tenant_succeeded_tasksCounterThe total number of tasks that succeeded
hatchet_tenant_failed_tasksCounterThe total number of tasks that failed (in a final state, not including retries)
hatchet_tenant_skipped_tasksCounterThe total number of tasks that were skipped
hatchet_tenant_cancelled_tasksCounterThe total number of tasks cancelled
hatchet_tenant_assigned_tasksCounterThe total number of tasks assigned to a worker
hatchet_tenant_scheduling_timed_outCounterThe total number of tasks that timed out while waiting to be scheduled
hatchet_tenant_rate_limitedCounterThe total number of tasks that were rate limited
hatchet_tenant_queued_to_assignedCounterThe total number of unique tasks that were queued and later got assigned to a worker
hatchet_tenant_queued_to_assigned_time_secondsHistogramBuckets of time in seconds spent in the queue before being assigned to a worker
hatchet_tenant_reassigned_tasksCounterThe total number of tasks that were reassigned to a worker
hatchet_tenant_used_worker_slotsGaugeThe current number of worker slots being used
hatchet_tenant_available_worker_slotsGaugeThe current number of worker slots available (free)
hatchet_tenant_worker_slotsGaugeThe total number of worker slots (free + used)
hatchet_tenant_used_worker_label_slotsGaugeThe current number of worker slots being used, by worker label pair and slot type
hatchet_tenant_available_worker_label_slotsGaugeThe current number of free worker slots, by worker label pair and slot type
hatchet_tenant_worker_label_slotsGaugeThe total number of worker slots (free + used), by worker label pair and slot type

The hatchet_tenant_*_worker_label_slots metrics expose gauges for each unique worker label (key, value) pair and slot type, using the label_key, label_value, and slot_type Prometheus labels. A worker’s slots count towards every label pair the worker carries, so a worker labeled pool=gpu, region=us-east contributes its slots to both the {label_key="pool", label_value="gpu"} and {label_key="region", label_value="us-east"} series.

The slot_type label separates the worker’s slot pools (e.g. default and durable), which have independent capacities. Filter to the slot type you care about — usually default — when computing utilization; a worker’s large durable slot pool would otherwise mask saturation of its default slots. Summing across slot_type values is safe (the pools are disjoint), unlike summing across label_key values.

⚠️

Because a worker contributes to one series per label key, summing these metrics across different label_key values counts the same slots multiple times. Always filter to a single (label_key, label_value) pair when querying.

Example PromQL Queries

1. Workflow Duration by Tenant and Status
rate(hatchet_tenant_workflow_duration_milliseconds_sum[5m])
by (tenant_id, workflow_name, status)
/
rate(hatchet_tenant_workflow_duration_milliseconds_count[5m])
by (tenant_id, workflow_name, status)
2. Tenant Queue Performance (95th percentile)
histogram_quantile(0.95,
  rate(hatchet_tenant_queued_to_assigned_time_seconds_bucket[5m])
) by (tenant_id)
3. Tenant Error Rate by Workflow
rate(hatchet_tenant_failed_tasks[5m]) by (tenant_id)
/
rate(hatchet_tenant_created_tasks[5m]) by (tenant_id)
4. Tenant Task Throughput
rate(hatchet_tenant_succeeded_tasks[5m]) by (tenant_id)
5. Tenant Retry Rate
rate(hatchet_tenant_retried_tasks[5m]) by (tenant_id)
/
rate(hatchet_tenant_created_tasks[5m]) by (tenant_id)
6. Workflow Duration Distribution by Tenant
sum by (tenant_id, le) (
  rate(hatchet_tenant_workflow_duration_milliseconds_bucket[5m])
)
7. Tenant Rate Limiting Impact
rate(hatchet_tenant_rate_limited[5m]) by (tenant_id)
8. Per-Tenant Queue Utilization
rate(hatchet_tenant_queue_invocations[5m]) by (tenant_id)
9. Tenant Scheduling Timeouts
rate(hatchet_tenant_scheduling_timed_out[5m]) by (tenant_id)
10. Tenant Task Assignment Success Rate
rate(hatchet_tenant_assigned_tasks[5m]) by (tenant_id)
/
rate(hatchet_tenant_created_tasks[5m]) by (tenant_id)
11. Tenant Task Reassignment Rate
rate(hatchet_tenant_reassigned_tasks[5m]) by (tenant_id)
12. Worker Slot Utilization by Label Pair
hatchet_tenant_used_worker_label_slots{label_key="pool", label_value="gpu", slot_type="default"}
/
hatchet_tenant_worker_label_slots{label_key="pool", label_value="gpu", slot_type="default"}

The worker slot gauges are designed to drive autoscalers — see Autoscaling Workers for how to use them with KEDA.

Cross-Tenant Analysis

Example PromQL Queries

1. Top 5 Tenants by Task Volume
topk(5,
  sum by (tenant_id) (
    rate(hatchet_tenant_created_tasks[1h])
  )
)
2. Slowest Workflows Across All Tenants
topk(10,
  rate(hatchet_tenant_workflow_duration_milliseconds_sum[5m])
  /
  rate(hatchet_tenant_workflow_duration_milliseconds_count[5m])
) by (tenant_id, workflow_name)
3. Tenant Resource Consumption Comparison
sum by (tenant_id) (
  rate(hatchet_tenant_workflow_duration_milliseconds_sum[1h])
)
/ 1000 / 60  # Convert to minutes

Integration with Prometheus

This endpoint can be used to configure Prometheus to scrape tenant-specific metrics:

scrape_configs:
  - job_name: "hatchet-tenant-metrics"
    static_configs:
      - targets: ["cloud.onhatchet.run"]
    metrics_path: "/api/v1/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/prometheus-metrics"
    scheme: "https"
    authorization:
      credentials: "your-api-token-here"

Note: Replace cloud.onhatchet.run with the URL where your Hatchet instance is hosted.

This provides tenant-isolated metrics that can be scraped directly by Prometheus or consumed by other monitoring tools that support the Prometheus text format.

Using PgBouncerInternal OpenTelemetry Traces