Parallelization

Parallelization is a pattern that runs several LLM calls at the same time and combines their outputs. A splitter fans the work out, independent workers process their pieces concurrently, and an aggregator fans the results back in. The total time is close to the slowest single call plus the merge step, rather than the sum of all the calls.

At a Glance

• Problem: Running independent checks one after another makes the user wait for every call in sequence, and a single call gives one sample of the model's judgment, which may be wrong and often covers several concerns thinly.
• Forces: Lower latency versus input tokens repeated to every worker; independent workers versus outputs that must stay consistent with each other; wide fan-out versus provider rate limits that all the calls hit at once.
• Solution: A splitter fans work out to independent LLM calls with no shared state, which run concurrently. An aggregator merges their outputs, either by combining distinct sections or by voting across repeated samples of the same task.
• Consequences: Gains: wall-clock time near the slowest worker plus the merge, short focused prompts, and a confidence signal from voting. Costs: higher token spend, concurrency code for timeouts, retries, and partial failures, and an aggregator that must reconcile conflicting findings.

What problem does Parallelization solve?

Prompt Chaining runs steps one after another, which is right when each step needs the previous one's output. Many tasks aren't like that. Reviewing a contract for liability, payment terms, and termination clauses involves three checks that don't depend on each other. Running them in sequence makes the user wait for three model calls when they could wait for one.

There is a second problem that parallelism solves. A single call gives you a single sample of the model's judgment, and that sample may be wrong. Asking the same question several times, or asking several differently-prompted reviewers, and comparing the answers often gives a more reliable result, along with a measure of how much the answers agree.

A third benefit is focus. A prompt asked to check many things at once tends to cover some of them thinly. Separate prompts that each check one thing often do better, because each call has a short, specific instruction and its whole context for a single concern. Anthropic makes the same observation about guardrails and evaluations, which work better as separate calls than folded into one.

How does Parallelization work?

Every parallel workflow has three parts.

The splitter decides what runs in parallel. Usually this is plain code: a fixed list of checks, one call per document chunk, or one call per item in a list. It can also be a model call that breaks a request into independent subtasks, but only when the subtasks can't be known in advance.

The workers are independent LLM calls with no shared state. Each gets its own prompt and its own slice of the input. Because they don't talk to each other, they can run concurrently and fail independently.

The aggregator merges the results. The merge strategy depends on what kind of parallelism you used:

• Sectioning splits a task into different subtasks: check security, check style, check tests. The aggregator concatenates or summarizes the sections, and every worker's output matters.
• Voting runs the same task several times, with different samples or different prompts. The aggregator takes a majority, an average, or the most conservative answer. Disagreement between workers is itself a useful signal. Self-Consistency is the voting form applied to reasoning.

The aggregator can be code (a vote, a merge, a threshold) or another model call that synthesizes the outputs. Prefer code when the merge rule is simple, since it is cheaper and deterministic.

When should you use Parallelization?

Use it when the work splits into pieces that don't need each other's results:

• Several independent checks or extractions run on the same input
• A long document or a list of items is processed one chunk or item at a time
• You want several independent judgments, for example from a guardrail or an evaluator, to reduce the chance that one bad sample decides the outcome
• Latency matters more than cost, and the sequential version is too slow

When not to use it: when steps depend on each other's output, when the pieces need to see each other to be consistent (such as sections of one essay that must not repeat each other), or when you are already near your provider's rate limit. Parallel calls hit rate limits all at once.

Implementation

# Using OpenAI SDK for illustration — swap client for any provider
import asyncio
from openai import AsyncOpenAI

MAX_CONCURRENT = 5   # stay under your provider's rate limit

CHECKS = {
    "liability": "List any clauses that create unlimited liability.",
    "payment": "List payment terms: amounts, due dates, late fees.",
    "termination": "List conditions under which either party can terminate.",
}

async def worker(client: AsyncOpenAI, limit: asyncio.Semaphore,
                 instruction: str, document: str) -> str:
    async with limit:   # waits here when MAX_CONCURRENT calls are in flight
        response = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "system", "content": instruction},
                      {"role": "user", "content": document}],
        )
    return response.choices[0].message.content or ""

async def review_contract(document: str) -> dict[str, str]:
    # Per-request timeout and SDK retries with backoff; created per run so
    # nothing is bound to an old event loop
    client = AsyncOpenAI(timeout=30, max_retries=2)
    limit = asyncio.Semaphore(MAX_CONCURRENT)
    tasks = [worker(client, limit, instr, document) for instr in CHECKS.values()]
    results = await asyncio.gather(*tasks, return_exceptions=True)   # fan out
    return {                                                          # fan in
        name: f"FAILED: {r!r}" if isinstance(r, BaseException) else r
        for name, r in zip(CHECKS, results)
    }

# Usage
# report = asyncio.run(review_contract(open("contract.txt").read()))

Three details keep this from being a toy. The semaphore caps how many calls are in flight. With three checks it never engages, but the same code processing a list of 200 items would otherwise fire 200 requests at once. The client's per-request timeout and retries with backoff stop one slow or rate-limited call from stalling the whole result. And return_exceptions=True means one failed worker produces a marked gap in the report instead of discarding the other results. The client and semaphore are created inside review_contract so the function can safely be called from separate asyncio.run invocations.

Worked Example: Cost and Latency

A contract review service runs the three checks above (liability, payment, termination) on 5,000 contracts a day using the small model.

Assumptions: a 6,000-token contract, a 100-token instruction per check (200 for the combined prompt), and 300 output tokens per check. The aggregator is code, so it adds no model call.

Each worker reads 6,100 tokens × $0.15 per million = $0.000915 and writes 300 × $0.60 per million = $0.00018, so $0.001095. Three workers cost $0.003285, or $3.29 per 1,000. The combined prompt reads the contract once: $0.00093 + $0.00054 = $0.00147 per contract, or $1.47 per 1,000. Parallel latency is the slowest worker, 0.5 + 300 / 150 = 2.5 s, against 3 × 2.5 = 7.5 s in sequence and 0.5 + 900 / 150 = 6.5 s for one long answer.

Parallel checks cost 2.2 times the combined prompt because the contract is sent three times, in exchange for 2.6 times lower latency; when latency does not matter and one prompt covers the checks well, the combined call is the cheaper choice.

What are the common pitfalls?

Hidden dependencies. Two subtasks that look independent sometimes aren't. If the "summary" worker and the "key risks" worker both run on the raw document, the summary may omit a risk the other worker flagged. When outputs must agree with each other, add a reconciliation step or run those parts in sequence.

Rate-limit storms. Fanning out 50 calls at once can trip your provider's requests-per-minute or tokens-per-minute limit, and the retries then arrive together as well. Bound concurrency and use retries with jittered backoff.

All-or-nothing failure. With a plain gather, one exception discards every result. Decide in advance whether a missing piece makes the whole answer invalid or just incomplete, and say so in the output.

A weak aggregator. Parallel workers often produce overlapping or contradictory findings. Concatenating them can hand the user a repetitive, inconsistent report. Deduplicate, and when workers disagree, show the disagreement rather than hiding it.

Voting without normalizing. A majority vote needs comparable answers. Extract the final answer from each worker in a fixed format before counting, and make the workers diverse enough not to share the same mistake. Self-Consistency covers both in depth.

What are the trade-offs?

Parallelization trades cost for latency. Sectioning sends the shared input to every worker, so input tokens grow with the number of workers while output tokens stay about the same. With a long document, that repetition dominates the bill. Prompt Caching on the shared prefix reduces it. Voting multiplies the whole cost by the number of votes, and its latency is set by the slowest of the N samples, not the typical one.

It also adds concurrency code: timeouts, partial failures, retries, and rate limits. That is ordinary engineering, but it is more than a single call needs.

In return, wall-clock time drops to about the slowest worker plus the merge, each prompt stays short and focused, and voting gives you a confidence signal you can't get from one sample.

Goes Well With

Self-Consistency is parallel voting applied to reasoning: sample several chains of thought and take the majority answer.

Prompt Chaining combines naturally with parallelization. A chain can fan out at one step and fan back in at the next, giving you a workflow that is sequential where it must be and parallel where it can be.

Multi-Agent Collaboration uses the same fan-out at the agent level, where each worker is an agent with its own tools rather than a single call.

LLM-as-Judge often serves as the aggregator for voting, scoring candidate outputs and picking the best.

References

• Anthropic. (2024). Building Effective Agents. Describes parallelization as a workflow with two variants, sectioning and voting.
• OpenAI. Function calling documentation. Parallel tool calls in a single model response.

Further Reading

• Kim et al., "An LLM Compiler for Parallel Function Calling" (2023) — Plans a graph of tool calls and runs independent ones in parallel, reporting lower latency and cost than sequential ReAct-style execution. Published at ICML 2024. arXiv:2312.04511
• Ning et al., "Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation" (2023) — Generates an answer outline first, then expands each point in parallel to speed up long responses, at some cost to coherence on some question types. arXiv:2307.15337