Troubleshooting Hatchet Workers

This guide aims to document common issues when deploying Hatchet workers.

Could not send task to worker

If you see this error in the event history of a task, it could mean several things:

1. The worker is closing its network connection while the task is being sent. This could be caused by the worker crashing or going offline.

2. The payload is too large for the worker to accept or the Hatchet engine to send. The default maximum payload size is 4MB. Consider reducing the size of the input data or output data of your tasks.

3. The worker has a large backlog of tasks in-flight on the network connection and is rejecting new tasks. This can occur if workers are geographically distant from the Hatchet engine or if there are network issues causing delays. Hatchet Cloud runs by default in us-west-2 (Oregon, USA), so consider deploying your workers in a region close to that for the best performance.

   If you are self-hosting, you can increase the maximum backlog size via the SERVER_GRPC_WORKER_STREAM_MAX_BACKLOG_SIZE environment variable in your Hatchet engine configuration. The default is 20.

No workers visible in dashboard

If you have deployed workers but they are not visible in the Hatchet dashboard, it is likely that:

1. Your API token is invalid or incorrect. Ensure that the token you are using to start the worker matches the token generated in the Hatchet dashboard for your tenant.

2. Worker heartbeats are not reaching the Hatchet engine. You will see noisy logs in the worker output if this is the case.

Phantom workers active in dashboard

This is often due to workers still running in your deployed environment. We see this most often with very long termination periods for workers, or in local development environments where worker processes are leaking. If you are in a local development environment, you can usually view running Hatchet worker processes via ps -a | grep worker (or whatever your entrypoint binary is called) and kill them manually.