Troubleshooting Serverless Endpoints

Start on the endpoint's page in the dashboard. Its status shows whether Hatchet can reach the endpoint, and Events shows what happened over the last few days.

Endpoint status

Status	Meaning
Healthy	The last healthcheck succeeded and its workflows were registered.
Unhealthy	Three healthchecks in a row failed, or the engine refused the endpoint's workflows. The status error says why.
Waiting for first healthcheck	The operator hasn't finished polling the endpoint yet. This should clear within one poll interval (30 seconds by default).
Disabled	The endpoint is disabled, so Hatchet neither polls it nor delivers to it.

One successful healthcheck makes an unhealthy endpoint healthy again. Health only reflects healthchecks: failed task deliveries don't change it.

An unhealthy endpoint still receives tasks when no healthy endpoint serves the same action, so a stale status never strands a task. When a healthy endpoint does serve it, the unhealthy one gets none.

An endpoint that stays Waiting for first healthcheck for more than a few poll intervals, or the status error no token for tenant, is a problem on the Hatchet side, not with your endpoint. On Hatchet Cloud, contact support. On a self-hosted instance, the in-engine operator is off by default: it runs when SERVER_SERVERLESS_OPERATOR_ENABLED is true, and only for tenants that have the serverless operator entitlement. Check the engine's logs for serverless operator errors.

Healthcheck stages

Each failed healthcheck records how far it got. The stage appears on HEALTHCHECK_FAILED and UNHEALTHY events.

REQUEST

No HTTP response came back. Common causes:

DNS. The hostname doesn't resolve. Check the URL for typos and that the deployment exists.
TLS. The host doesn't serve a valid, publicly trusted certificate. A new custom domain may not have one yet.
Connection. The connection was refused or dropped. Make sure the deployment is running and serves HTTPS on port 443.
Timeout. The endpoint didn't answer within the healthcheck timeout, 10 seconds by default. Keep the healthcheck fast, and avoid slow work at cold start.
Blocked destination. The URL isn't https, names a port other than 443, or resolves to a loopback or private address. Hatchet refuses these. For local development, use a public HTTPS tunnel.
Signing secret. Hatchet couldn't load the endpoint's signing secret. Rotate it from Edit and set the same value on the endpoint.

STATUS

The endpoint answered with a status outside 2xx. The response excerpt usually says why.

401 or 403. The signature check failed or something in front of the endpoint refused the request:
- The endpoint's signing secret doesn't match the one registered in Hatchet. If you aren't sure, rotate it and set the new value on both sides.
- The endpoint's clock is more than 5 minutes off.
- Cloudflare Access, WAF or bot rules, or Vercel Deployment Protection, answered before your code ran. See the Cloudflare and Vercel notes.
404 or 405. The host answered, but no Hatchet handler runs at that path or for POST. Check the healthcheck URL against the routes your handler serves.
3xx. Hatchet doesn't follow redirects. Register the final URL, for example with or without a trailing slash, or on the canonical domain.
5xx. The handler crashed or the platform failed. Check the platform's logs.

PARSE

The endpoint answered 2xx, but the body isn't a healthcheck response. This is typically an HTML page, an empty body, or JSON with fields of the wrong type. See Healthcheck for the expected shape.

Unknown fields are ignored rather than rejected. A misspelt workflows field therefore doesn't fail at this stage: the endpoint turns healthy with no actions. If an endpoint is healthy but registered nothing, check the field names.

CATALOG

The response parsed, but the workflows or actions it declares were refused:

More than 200 workflows or 500 actions (the default limits).
An action id that's empty, isn't of the form <workflow>:<task>, contains control characters, or is longer than 1024 bytes.
A tasks entry that names an action the endpoint doesn't serve.
The engine refused to register a workflow, for example because its definition is invalid. This records a CATALOG_REJECTED event and marks the endpoint unhealthy right away. Hatchet retries the same catalog with a growing backoff, and a changed catalog is tried on the next poll.

Delivery failures

A failed task delivery fails the task run with an error message that says why. The task's retry policy applies only to failures Hatchet marks retryable. See the full status mapping.

Task error	Cause
`request timed out after …`	The endpoint didn't answer within its request timeout. Raise it (up to 600s), within your platform's own limit. Retried.
`endpoint returned status 5xx …`	The handler failed. Retried.
`endpoint returned status 4xx …`	The endpoint refused the task. Not retried, except `408`, `425` and `429`.
`endpoint returned redirect status 3xx; redirects are not followed`	The trigger URL redirects. Register the final URL.
`endpoint returned status 2xx with a non-JSON body`	A `2xx` response must be JSON (or empty). Not retried.
`response body exceeded the 4 MiB limit`	The output is too large. Not retried.
`request failed: …`	A connection or TLS error. Retried.
Your own message	The endpoint answered with `{"error": "...", "retry": ...}`.

Each failure also records a DELIVERY_FAILED event on the endpoint, with the action, task run, retry count, status code and the start of the response body.

Endpoints built with @hatchet-dev/serverless answer with their own messages for the common cases:

Task error	Cause
`bad signature`, or a timestamp outside the window	The signing secret on the endpoint doesn't match Hatchet's, or the clocks differ by more than 5 minutes. Not retried.
`… configure \`client` to enable this`	The task called `ctx.runChild`, `ctx.putStream` or another member that needs a Hatchet client, and the endpoint has no `HATCHET_CLIENT_TOKEN`. Not retried.
`… invoked over a POST`	The task waited for a child run or read a stream, but was delivered as a `POST` because the endpoint doesn't accept the websocket. On Vercel, see platform notes. Not retried.
`this endpoint does not serve durable tasks`	A websocket upgrade reached an endpoint that can't accept it: no `@vercel/functions` and `ws`, no Fluid compute, or `durable: false`. Not retried.
`nondeterminism …`	A durable task replayed differently from its recorded log after an eviction. The task must be deterministic. Not retried.

Events

The Events log on the endpoint's page lists what the operator recorded, newest first. Events are kept for 7 days. You can also read them with GET /api/v1/stable/serverless/endpoints/{endpoint}/events (limit defaults to 100, at most 500, and since filters by time) or the SDK calls in Managing endpoints.

Event	Recorded when
`HEALTHCHECK_FAILED`	A healthcheck failed. Recorded on the first failure of a streak and whenever the error changes, not on every poll. Details include the stage, URL, status code, duration and response excerpt.
`UNHEALTHY`	The endpoint became unhealthy, or became unhealthy for a new reason.
`HEALTHY`	The endpoint became healthy.
`CATALOG_REGISTERED`	A changed set of workflows and actions was registered.
`CATALOG_REJECTED`	The engine refused the declared workflows.
`DELIVERY_FAILED`	A task delivery failed. Recorded at most once every 30 seconds per endpoint. `details.suppressed` counts the failures in between that weren't recorded on their own.

Response excerpts on events are cut to 1 KiB.

Events are diagnostics. If the database is slow, Hatchet may drop events rather than delay polling or delivery, so a gap in the log doesn't prove nothing happened.

We use cookies