Orchestrator-Workers

Orchestrator-Workers is a pattern that lets a model decide how to split a task. An orchestrator LLM reads the request, writes a list of subtasks that could not have been known in advance, and hands each one to a worker that starts with a clean, focused context. The workers run concurrently and return short reports, and a synthesis step combines them into one answer.

At a Glance

• Problem: Some tasks split into parallel pieces, but which pieces depends on the input. A fixed fan-out cannot anticipate them, and a single agent doing every piece fills its context with material from the others.
• Forces: Flexible decomposition versus predictable cost and behavior; clean worker contexts versus shared decisions the workers never see; compressed reports versus detail lost before synthesis; wide fan-out versus rate limits and token spend.
• Solution: An orchestrator returns a JSON list of subtasks, which code validates and caps. Workers run concurrently with a timeout each, failures are marked rather than dropped, and a synthesizer merges the short reports, naming any gaps.
• Consequences: Gains: decomposition that adapts to each input, focused worker contexts, cheap models on the heavy reading, and wall-clock time near the slowest worker. Costs: two extra large-model calls, variable spend per request, and workers that cannot coordinate.

What problem does Orchestrator-Workers solve?

Parallelization works when your code knows the split in advance: three fixed contract checks, one call per chunk, one call per item. Many useful tasks do not look like that. "Compare the data-retention terms of the vendors on this shortlist" needs one subtask per vendor, and the shortlist changes with every request. "Update every call site affected by this API change" needs one subtask per file, and you only learn which files after reading the change. "What do regulators in our markets say about this?" depends on which markets the question names. The number and the content of the subtasks are part of the answer.

You could hand the whole task to one agent. That agent reads vendor A's policy, then vendor B's, then C's, and by the time it reaches E its context holds four other documents, the notes it took on each, and its earlier reasoning. Every later step pays to re-read all of that, and the model's attention is spread across material that has nothing to do with the current step. Running the pieces one after another also makes the user wait for every one of them in turn.

Orchestrator-Workers separates the two jobs that single agent was doing. One model call decides what the pieces are. Separate workers each handle one piece with only what that piece needs, and they run at the same time. The orchestrator never reads the raw material; it sees only the plan and, later, the workers' compressed findings.

How does Orchestrator-Workers work?

The pattern has four parts, and the boundaries between them matter more than any prompt.

The orchestrator is an LLM call that turns the request into a list of subtasks. Ask for structured output, a JSON object with a subtasks array, and treat the reply as untrusted input. Code parses it, drops items without a usable instruction, and cuts the list to a hard maximum. If the plan can't be parsed at all, fall back to something safe, such as answering with a single call, instead of guessing at the model's intent. The orchestrator should also be told to use as few subtasks as the task needs. Anthropic reported that early versions of its research system spawned far too many subagents for simple queries, and fixed it with explicit scaling rules in the lead agent's prompt.

The workers each receive the overall goal and one subtask, nothing else. This is the point of the pattern. A worker reading vendor C's policy does not see vendors A, B, D, or E, so its context is short and on topic. Anthropic's write-up describes what a good delegation contains: an objective, an output format, guidance on which tools and sources to use, and clear boundaries so workers don't duplicate each other. Workers can be single calls, as in the code below, or full agents running a ReAct Loop with their own tools.

The bounds are enforced in code, not in the prompt. A cap on subtasks limits spend per request. A semaphore limits how many workers are in flight. A per-worker timeout stops one slow call from holding up the rest. A round limit applies when the orchestrator is allowed to look at the reports and dispatch a second wave: two or three rounds, then synthesize what you have.

The synthesizer receives the workers' reports, not their full transcripts, and writes the final answer. Workers should return compressed findings in a fixed shape so the synthesizer's input stays small. A worker that failed or timed out still appears in that input, marked as failed, and the synthesizer is told to name the gap rather than fill it. Often the orchestrator model does the synthesis as well, since it already knows why each subtask exists.

How this differs from its neighbours:

• Parallelization fixes the split in code. Orchestrator-Workers asks a model for the split on each request. If you can write the list of subtasks yourself, do that instead.
• Plan-and-Execute also starts with an LLM-written plan, but one agent carries out the steps in order, and later steps can use earlier results. Orchestrator-Workers assumes the subtasks are independent and gives each to a separate worker with its own context.
• Multi-Agent Collaboration describes the broader family: specialist roles, supervisors, peer messaging, hierarchies. Orchestrator-Workers is the one supervisor shape where the workers are interchangeable and the delegation is decided at run time, which makes it the one-level form of the hierarchical topology in Multi-Agent Topologies. This page covers the mechanics of that decomposition and the synthesis that follows it.

When should you use Orchestrator-Workers?

• The task splits into independent pieces, but which pieces depends on the input
• Each piece needs a lot of reading (documents, search results, files) that the other pieces don't need
• The heavy reading can go to a smaller, cheaper model while a larger model plans and synthesizes
• Wall-clock time matters and the pieces can run at the same time
• Breadth-first research, where the question fans out into several directions that can be explored separately

When not to use it: if your code can enumerate the subtasks, use Parallelization; it is cheaper and deterministic. If the steps depend on each other's results, use Plan-and-Execute or Prompt Chaining. If the pieces must agree on many shared decisions, such as parts of one design or one codebase change that must fit together, a single agent that keeps the full context is usually safer, which is the argument Cognition makes against splitting such work across agents. And if one call with everything in context already answers well, the orchestration is overhead.

Implementation

# Using OpenAI SDK for illustration — swap client for any provider
import asyncio
import json
from openai import AsyncOpenAI

MAX_SUBTASKS = 6       # hard cap on fan-out, whatever the orchestrator asks for
MAX_CONCURRENT = 5     # workers in flight at once
WORKER_TIMEOUT = 60    # seconds per worker call
PLAN_PROMPT = ('Split the task into independent subtasks. Reply with JSON only: '
               '{"subtasks": [{"instruction": "..."}]}. Use as few as the task needs.')

async def ask(client: AsyncOpenAI, model: str, system: str, user: str) -> str:
    response = await client.chat.completions.create(model=model, messages=[
        {"role": "system", "content": system}, {"role": "user", "content": user}])
    return response.choices[0].message.content or ""

def parse_plan(raw: str) -> list[str]:
    try:
        items = json.loads(raw)["subtasks"]
    except (json.JSONDecodeError, KeyError, TypeError):
        return []                                   # malformed plan: no subtasks
    if not isinstance(items, list):
        return []
    instructions = [s.get("instruction") for s in items if isinstance(s, dict)]
    return [i.strip() for i in instructions if isinstance(i, str) and i.strip()][:MAX_SUBTASKS]

async def run_worker(client, limit, task: str, instruction: str) -> tuple[bool, str]:
    async with limit:
        try:
            result = await asyncio.wait_for(ask(client, "gpt-4o-mini",
                "Do only this subtask. Report findings in under 200 words.",
                f"Overall goal: {task}\nYour subtask: {instruction}"), WORKER_TIMEOUT)
            return True, result
        except Exception as exc:                    # timeout or API error
            return False, f"FAILED: {exc!r}"

async def orchestrate(task: str) -> str:
    client = AsyncOpenAI(timeout=60, max_retries=2)  # bounds plan and synthesis calls too
    plan = parse_plan(await ask(client, "gpt-4o", PLAN_PROMPT, task))
    if not plan:                                    # fall back to a single call
        return await ask(client, "gpt-4o", "Complete the task.", task)
    limit = asyncio.Semaphore(MAX_CONCURRENT)
    results = await asyncio.gather(*[run_worker(client, limit, task, i) for i in plan])
    if not any(ok for ok, _ in results):
        raise RuntimeError("every worker failed; nothing to synthesize")
    reports = "\n\n".join(f"## {i}\n{text}" for i, (_, text) in zip(plan, results))
    return await ask(client, "gpt-4o",
        "Combine the worker reports into one answer. Name any FAILED subtask "
        "and do not fill its gap by guessing.", f"Task: {task}\n\n{reports}")

# Usage
# answer = asyncio.run(orchestrate("Compare data-retention terms across our five vendors"))

The plan is parsed defensively: anything that isn't a list of objects with a non-empty instruction is dropped, the list is cut to MAX_SUBTASKS, and an unusable plan falls back to one direct call rather than raising or looping. Each worker sees only the goal and its own subtask, runs under the semaphore with its own wait_for timeout, and returns a success flag instead of raising, so one failure becomes a labelled FAILED entry in the synthesizer's input. If every worker fails there is nothing to synthesize, so the function raises instead of asking the model to invent an answer. This version runs one round. To allow follow-up rounds, wrap the plan, dispatch, and gather steps in a loop bounded by a MAX_ROUNDS constant, and pass the previous reports to the orchestrator on each pass.

In the worked example each instruction embeds or points to one policy; the snippet leaves document loading to your retrieval layer.

Worked Example: Cost and Latency

A procurement team asks free-form questions about its vendors, such as "compare data-retention and deletion terms across our shortlist." Each question touches five vendor policies of about 8,000 tokens each, and which five depends on the question. The comparison runs on the large model reading all five documents in one call, against orchestrator-workers with the large model planning and synthesizing and the small model as workers.

Assumptions: the single call reads 40,000 tokens of policy plus a 500-token prompt and writes 1,500 tokens. The orchestrator reads 600 tokens (question, vendor list, instructions) and writes a 250-token plan. Each of five workers reads one policy plus 200 tokens of instructions (8,200 tokens) and writes a 400-token report. The synthesizer reads the five reports plus 500 tokens of context (2,500 tokens) and writes 1,500. No caching.

The orchestrator costs 600 × $3.00 per million + 250 × $15.00 per million = $0.00555. Each small-model worker costs 8,200 × $0.15 per million + 400 × $0.60 per million = $0.00147, so five cost $0.00735. The synthesizer costs 2,500 × $3.00 per million + 1,500 × $15.00 per million = $0.03. The total is $0.0429 per request, against $0.144 for the single call (40,500 × $3.00 per million + 1,500 × $15.00 per million). Latency is the orchestrator (0.5 + 250 / 60 = 4.7 s), then the slowest worker (0.5 + 400 / 150 = 3.2 s), then the synthesizer (0.5 + 1,500 / 60 = 25.5 s), or 33.3 s. In sequence the workers take 5 × 3.2 s instead, for 46.0 s. With large-model workers each costs $0.0306 and takes 0.5 + 400 / 60 = 7.2 s.

The pattern costs 3.4 times less than the single call here because the 40,000 tokens of reading move to the small model, but it is 7.8 s slower, since the plan and the synthesis are two large-model calls on the critical path. Run with large-model workers it costs more than the single call and is slower too. The savings come from model choice, not from the orchestration itself, and the single call is no longer an option once the documents stop fitting in one context.

What are the common pitfalls?

Over-decomposition. Left alone, an orchestrator often splits a simple question into many subtasks, and each one costs a worker call plus tokens in the synthesizer's input. Cap the count in code, tell the orchestrator to scale effort to the question, and log the subtask count per request so you notice when it creeps up.

Thin delegations. A subtask like "research vendor C" leaves the worker to guess the scope, the output shape, and where to stop. Workers then overlap, miss the point, or return reports the synthesizer can't compare. Make the orchestrator state an objective, a return format, and a boundary for each subtask, and give workers a fixed report schema.

Hidden coupling between workers. Workers cannot see each other, so any decision one makes is invisible to the rest. Cognition's example is two subagents building parts of one game in visibly different styles. If the pieces must fit together, put the shared decisions in every worker's instructions, or don't split the work.

Synthesis that papers over gaps. A worker that timed out leaves a hole, and a synthesizer asked for "a complete comparison" will fill it with something plausible. Pass failures through explicitly, instruct the synthesizer to report them, and decide in advance how many failures make the whole answer invalid.

Unbounded rounds. Letting the orchestrator review the reports and dispatch more work is useful, but without a round limit and a budget it can keep finding one more thing to check. Bound the rounds in code and synthesize what you have when the limit is reached.

What are the trade-offs?

You gain decomposition that fits each input without writing the split yourself, worker contexts that stay short and relevant, the option to put the bulk of the reading on a small model, and wall-clock time near the slowest worker rather than the sum.

You pay with two large-model calls on every request, plan and synthesis, which add latency even when the work is small. Spend varies with the plan, so per-request cost is harder to predict than in a fixed workflow, and a cap is the only firm bound. Information is lost twice: once when the orchestrator writes a short instruction for each worker, and again when each worker compresses its findings. Debugging also gets harder, because a wrong answer may trace back to a bad plan, a bad worker, or a bad merge, so log all three. Anthropic reported that its multi-agent research system used many times more tokens than ordinary chat interactions, and concluded the approach pays off for broad, high-value tasks rather than everyday ones.

Goes Well With

Parallelization supplies the fan-out machinery. Once the orchestrator has produced a plan, the dispatch step is ordinary parallelization with a semaphore, timeouts, and partial-failure handling.

Agentic RAG is a natural worker. Each worker runs its own retrieve, read, and refine loop on one subtopic, and returns only its findings, which keeps the retrieved passages out of the orchestrator's context. Deep Search is the same idea applied to open-ended research.

Grounded Generation belongs in the synthesizer. Ask workers to return claims with sources, and have the synthesizer cite them, so the final answer can be traced back to the worker that produced each claim.

Model Router decides which model each role uses. The worked example shows the economics depend on putting workers on a small model and keeping the orchestrator and synthesizer on a large one.

References

• Anthropic. (2024). Building Effective Agents. Names orchestrator-workers as a workflow in which a central LLM breaks down tasks dynamically, delegates them to workers, and synthesizes their results, and contrasts it with parallelization's predefined subtasks.
• Anthropic. (2025). How we built our multi-agent research system. A lead agent plans and spawns subagents with separate context windows that search in parallel; covers delegation instructions, scaling effort to query complexity, and token cost.
• Cognition. (2025). Don't Build Multi-Agents. Argues that subagents working without shared context make conflicting decisions, and recommends single-threaded agents for tightly coupled work.
• OpenAI. OpenAI Agents SDK documentation: Handoffs and agents as tools. Framework support for one agent delegating to others.

Further Reading

• Shen et al., "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face" (2023) — An LLM controller plans subtasks from the request, dispatches each to a specialist model, and summarizes the results, an early instance of run-time decomposition with a synthesis step. arXiv:2303.17580
• Fourney et al., "Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks" (2024) — An orchestrator agent keeps a task ledger and a progress ledger, directs specialized agents, and replans when progress stalls. arXiv:2411.04468