We use cookies

We use cookies to ensure you get the best experience on our website. For more information on how we use cookies, please see our cookie policy.

By clicking "Accept", you agree to our use of cookies.
Learn more.

Bulk Run Many Tasks

Often you may want to run a task multiple times with different inputs. There is significant overhead (i.e. network roundtrips) to write the task, so if you’re running multiple tasks, it’s best to use the bulk run methods.

You can use the aio_run_many method to bulk run a task. This will return a list of results.

examples/child/bulk.py

greetings = ["Hello, World!", "Hello, Moon!", "Hello, Mars!"]

results = await child_task.aio_run_many(
    [
        # run each greeting as a task in parallel
        child_task.create_bulk_run_item(
            input=SimpleInput(message=greeting),
        )
        for greeting in greetings
    ]
)

# this will await all results and return a list of results
print(results)

Workflow.create_bulk_run_item is a typed helper to create the inputs for each task.

There are additional bulk methods available on the Workflow object.

aio_run_many
aio_run_many_no_wait

And blocking variants:

run_many
run_many_no_wait

As with the run methods, you can call bulk methods from within a task and the runs will be associated with the parent task in the dashboard.

You can use the run method directly to bulk run tasks by passing an array of inputs. This will return a list of results.

src/v1/examples/simple/bulk.ts

const res = await simple.run([
  {
    Message: 'HeLlO WoRlD',
  },
  {
    Message: 'Hello MoOn',
  },
]);

console.log(res[0].TransformedMessage);
console.log(res[1].TransformedMessage);

There are additional bulk methods available on the Task object.

run
runNoWait

As with the run methods, you can call bulk methods on the task fn context parameter within a task and the runs will be associated with the parent task in the dashboard.

src/v1/examples/simple/bulk.ts

const parent = hatchet.task({
  name: 'simple',
  fn: async (input: SimpleInput, ctx) => {
    // Bulk run two tasks in parallel
    const child = await ctx.bulkRunChildren([
      {
        workflow: simple,
        input: {
          Message: 'Hello, World!',
        },
      },
      {
        workflow: simple,
        input: {
          Message: 'Hello, Moon!',
        },
      },
    ]);

    return {
      TransformedMessage: `${child[0].TransformedMessage} ${child[1].TransformedMessage}`,
    };
  },
});

Available bulk methods on the Context object are: - bulkRunChildren - bulkRunChildrenNoWait

You can use the RunBulkNoWait method directly on the Task object to bulk run tasks by passing an array of inputs. This will return a list of run IDs.

examples/v1/run/bulk.go

simple := v1_workflows.Simple(hatchet)
bulkRunIds, err := simple.RunBulkNoWait(ctx, []v1_workflows.SimpleInput{
	{
		Message: "Hello, World!",
	},
	{
		Message: "Hello, Moon!",
	},
})

if err != nil {
	panic(err)
}

fmt.Println(bulkRunIds)

Additional bulk methods are coming soon for the Go SDK. Join our Discord to stay up to date.

Event Trigger Concurrency