Sticky Worker Assignment (Beta)

This feature is currently in beta and may be subject to change.

Sticky assignment is a workflow property that allows you to specify that all steps of a workflow should be assigned to the same worker for the duration of its execution. This can be useful in situations like when you need to maintain expensive local memory state across multiple steps in a workflow or ensure that certain workflows are processed by the same worker for consistency.

This feature is only compatible with long lived workers, and not webhook workers.

Setting Sticky Assignment

Sticky assignment is set on the workflow level by adding the sticky property to the workflow definition. When a workflow is marked as sticky, all steps within that workflow will be assigned to the same worker for the duration of the workflow execution.

⚠️

While sticky assignment can be useful in certain scenarios, it can also introduce potential bottlenecks if the assigned worker becomes unavailable, or if local state is not maintained when the job is picked up. Be sure to consider the implications of sticky assignment when designing your workflows and have a plan in place to handle local state issues.

There are two strategies for setting sticky assignment:

SOFT: The all steps in the workflow will attempt to be assigned to the same worker, but if that worker is unavailable, it will be assigned to another worker.
HARD: The all steps in the workflow will only be assigned to the same worker. If that worker is unavailable, the step run will not be assigned to another worker and will remain in a pending state until the original worker becomes available or timeout is reached. (See Scheduling Timeouts)

@hatchet.workflow(
    on_events=["user:create"],
    sticky=StickyStrategy.SOFT, # <-- specify the sticky strategy
)
class StickyWorkflow:
    @hatchet.step()
    def step1a(self, context: Context):
        return {"worker": context.worker.id()}
 
    @hatchet.step()
    def step1b(self, context: Context):
        return {"worker": context.worker.id()}
 
    @hatchet.step(parents=["step1a", "step1b"])
    def step2(self, context: Context):
        return {"worker": context.worker.id()}

In this example, the sticky property is set to SOFT, which means that the workflow will attempt to be assigned to the same worker for the duration of its execution. If the original worker is unavailable, the workflow will be assigned to another worker.

Sticky Child Workflows

It is possible to spawn child workflows on the same worker as the parent workflow by setting the sticky property to true in the spawnWorkflow method options. This can be useful when you need to maintain local state across multiple workflows or ensure that child workflows are processed by the same worker for consistency.

However, the child workflow must:

Specify a sticky strategy in the child workflow's definition
Be registered with the same worker as the parent workflow

If either condition is not met, an error will be thrown when the child workflow is spawned.

@hatchet.workflow(
    on_events=["sticky:parent"],
)
class StickyWorkflow:
    @hatchet.step(parents=["step1a", "step1b"])
    async def parent_step(self, context: Context):
        ref = context.spawn_workflow('StickyChildWorkflow', {}, options={"sticky": True})
        await ref.result()
        return {"worker": context.worker.id()}

@hatchet.workflow(
on_events=["sticky:child"],
sticky=StickyStrategy.SOFT
)
class StickyChildWorkflow:
@hatchet.step()
def child(self, context: Context):
return {"worker": context.worker.id()}

Overview Worker Affinity