Task Eviction
Since durable tasks are often waiting for sleeps, events, or child runs, they spend much of their time inactive. When this happens, the durable task no longer needs to hold a slot on the worker until the thing it’s waiting for completes. Hatchet has the option to evict durable tasks from workers when they’re in one of these waiting states, in order to free up slots on the worker. This allows the worker to pick up additional durable tasks, and then resume the original one from where it left off when the wait is satisfied, without keeping the idle task around.
It’s important to note that eviction is a fundamental difference between durable and non-durable tasks. Only durable tasks can be evicted from and restored on workers, as opposed to non-durable tasks, which will hold slots throughout the entirety of their execution, even while they’re inactive.
Configuring Eviction Policies
Every durable task can configure an eviction policy, which tells Hatchet how to respond when that task hits a wait. There are a few configuration options available to you to choose from:
- You can set a TTL, which tells Hatchet that the durable task should only be evicted after some amount of uninterrupted time has been spent waiting.
- You can enable or disable capacity-based eviction, which allows Hatchet to optimistically evict the durable task if the worker is running out of slots and wants to pick up a new durable task run, but would be unable to if the waiting task could not be evicted.
- Finally, you can provide a priority, which tells Hatchet what order candidate durable tasks should be evicted in (lower priority values will be evicted first when choosing between multiple candidate durable tasks to evict).
Task Resumption
Hatchet will resume a task in an evicted state when the thing it was waiting for is satisfied. When this happens, the durable task is re-triggered on the worker, and its event log is replayed up to the checkpoint where it left off, at which point it continues making progress.
Since durable tasks can be evicted or resumed, it’s important to make sure that a durable task doesn’t perform any operations that you wouldn’t want to occur multiple times. For instance, non-idempotent writes to a database, sending emails, or computationally expensive work are generally not good candidates for the kinds of things to do in a durable task, since that logic will be rerun every time the task is resumed.