Prompt Dependency Graphs That Shrink Your Retest Set

A two-line edit to a planner prompt ships on a Friday afternoon. The diff looks harmless: tighten the step format, rename one field. By Tuesday the tool caller two hops downstream is selecting the wrong tools, and it takes a customer complaint before anyone connects the two events. The regression lived not in the changed prompt but in the coupling between prompts, and nothing in the review process was built to see that coupling.

In a multi-prompt LLM system, prompts are not independent text files. They are components wired together by shared context blocks, parsed outputs, retrieved examples, and tool contracts, which makes every prompt edit a change-management event rather than a wording tweak. A prompt dependency graph treats it that way. Nodes are your prompts plus the components that consume them, edges are the mechanisms that couple them, and the graph turns the question of which evals to rerun when a prompt changes from a gut call into a computation.

Incident reviews of these failures share a shape. The diff was read, approved, and genuinely fine, because the defect sat in a consumer the diff never touched. In the prompt-change regressions teams hit in production, the broken component usually sits downstream of the edit, and untracked coupling is the mechanism. Reviewing the changed text harder can never catch what lives outside it.

Without a way to see coupling, teams oscillate between two failure modes. Rerun every eval and the suite takes hours, so people quietly start skipping it. Rerun nothing and you ship on hope. A widely shared anecdote about one prompt change rippling into dozens of others surfaced this pain from a single codebase; the goal here is a playbook any team can run: the five edge types that do most of the coupling, a method to rebuild the graph from LLM observability traces you already collect, a two-step selection rule, and a worked example you can copy.

Five dependency edges that couple prompts in production

Prompt-to-prompt coupling is not exotic. In the agent pipelines we have inspected, five mechanisms account for most of it, and each maps to fields a trace already records.

The ranking blends frequency with detectability. Schema edges fail hard and fast, so they rank first even though they are also the easiest to catch with a conformance check. Shared-block edges are the sneakiest: shared system prompt coupling risk scales with the number of consumers, and the blast radius of a persona edit is every component that inherits it. Tool-description coupling runs in both directions, since a prompt that references tools depends on descriptions it does not contain, and the function calling schemas your tools expose get consumed by prompts nobody thinks of as dependent.

Few-shot drift deserves its own edge because it couples indirectly. When examples are retrieved at runtime, an upstream change can alter what gets retrieved and shift a downstream prompt's behavior with zero edits to any prompt file. It returns in the failure modes below, because it is also the edge most likely to defeat a naive graph.

Rebuilding the prompt dependency graph from traces you already log

You rarely need new instrumentation to reconstruct prompt dependencies from traces. Structured spans from LLM observability tooling already carry the signals, and OpenTelemetry's GenAI semantic conventions standardize attributes for prompts, models, token counts, and tool calls, so the raw material is usually sitting in your telemetry store today.

Derive the edges with five joins:

1. Name the nodes. Give every prompt a stable ID plus a version hash, and make both mandatory span attributes. Without version hashes you cannot tell which edges belonged to which revision.
2. Join outputs to inputs. Within a trace, when prompt A's output schema matches prompt B's input schema, record a schema edge. Repeated co-occurrence across traces confirms the consumer relationship and filters one-off coincidences.
3. Match tool calls to descriptions. A tool name invoked in one component's span, against that tool's description living in another prompt's tool block, yields a tool edge.
4. Hash shared blocks. Hash system-prompt segments, not whole files. Partial sharing, where two prompts reuse one paragraph of a persona block, is common and invisible to file-level hashing.
5. Watch the budgets. Track input token counts per component. When a component's input length distribution shifts after an upstream version bump, add a context-budget edge.

This is an afternoon of SQL or pandas over a few thousand traces, not a platform project. The result is approximate, and it should be: verify the highest-risk edges by reading the consuming code, and let the rest stay inferred. An approximate graph that is written down beats a perfect mental model that lives in one engineer's head.

From blast radius to minimal retest set

The blast radius is a property of the graph. The retest set is a property of the graph plus your eval suite and your risk tolerance.

This distinction is the whole method. When teams conflate the two lists, they either retest everything the change can reach, which is how a five-minute edit can trigger a two-hour pipeline, or they skip retesting entirely and ship on hope. Keep the lists separate and both failure modes disappear.

Step 1, compute the blast radius. Take the changed prompt as the source node, walk the dependency edges transitively, and collect everything reachable, including second- and third-hop consumers. This is how you compute the blast radius of a prompt change instead of guessing at it.

Step 2, filter down to the retest set. For each reached component, ask three questions. Which edge class connects it, and how risky is that class? Does an eval exist that exercises the coupled behavior? Would that eval actually catch the failure mode the edge produces? Retest only where the answers line up, which is why selection also depends on evals that catch regressions rather than demos that pass.

Publish both lists in the release note. The reach list tells reviewers what you considered; the retest list tells them what you promise to check. Graph-scoped prompt regression testing this way typically shrinks a full-suite rerun to a handful of evals, and the shrinkage is defensible because the reasoning is written next to it.

A worked example with a five-component agent pipeline

Picture a support agent with five components: a router that classifies the request, a planner that emits a JSON plan, a retriever that searches the knowledge base from step descriptions, a tool caller that executes plan steps, and a synthesizer that writes the final answer from tool output and retrieved documents. The router, planner, and tool caller share a persona block in their system prompts.

The change: planner v12 becomes v13, renaming the plan field note to hint and tightening step formatting inside the planner-local section. Because prompts are stored whole, the shared persona block sits inside the edited file, so the graph conservatively fires shared-block edges too. Seven candidates come out of the traversal.

Three evals cover seven candidates, and every cut carries written reasoning: the shared-block hits are low risk because the shared segment itself did not change, the retriever edge fired on a schema heuristic that a trace sample disproved, and the synthesizer's two rows collapse into one run. If a cut feels uncomfortable, the fallback is a cheap smoke eval on the cut component, not a full rerun, which keeps the gate fast while hedging the judgment call.

Why versioning and A/B testing stop short

None of the standard prompt tooling answers the scoping question, because none of it models dependencies between prompts at all.

Registries such as LangSmith prompt versioning make the diff question easy, and that matters, since you cannot scope what you cannot see. But the limitations of prompt versioning appear the moment one prompt consumes another's output: a perfectly recorded diff of prompt A says nothing about prompt B's parser. A/B tests fail differently. They measure average user-facing outcomes, and averages hide distribution shifts, so a broken intermediate component can hide inside a winning variant. Use versioning to know what changed, A/B to choose between candidates, and the graph to scope your LLM eval reruns. The three compose; none substitutes for another.

Failure modes that break naive dependency graphs

A graph built from traces is a model, and models lie in specific ways. Four failure modes account for most of the lying.

1. Dynamically retrieved few-shot examples. Few-shot drift in agent pipelines couples components that share no code and no prompt text. An upstream change alters retrieval inputs, the pool returns different examples, and downstream behavior shifts. Mitigation: log the selected example IDs per call and model the pool itself as a graph node.
2. Response and context caches. A cache hit during the retest window returns completions from before the change, so the eval passes against stale behavior. Mitigation: include the prompt version in cache keys, or bypass the cache for blast-radius evals.
3. Model-version swaps. The same prompt hash over a new model behaves differently, and a graph keyed only on prompt identity sees no change at all. Mitigation: make the model version part of node identity and treat a swap as a change event with its own radius.
4. Silent schema drift. Tolerant parsers absorb malformed output until something far downstream breaks, which means the trace data your edges derive from is itself misleading. Mitigation: assert schema conformance in traces and alert on drift, so the graph learns from real consumption rather than accommodated breakage.

A change-management checklist for prompt releases

LLM prompt change management reduces to five steps, and every one is mechanical once the graph exists.

1. Diff the change against the graph. Which prompt changed, to which version, and which edge types does the diff touch?
2. Compute both lists. Transitive closure gives the reach set; risk-class and eval-coverage filtering gives the retest set.
3. Gate the release on the retest set. Block the merge until those evals pass, wired into your harness, whether homegrown or built on the OpenAI evals framework.
4. Stage the rollout. Canary traffic first, with the blast-radius components watched most closely.
5. Validate the graph post-deploy. Compare trace distributions, input schemas, output drift, and token budgets on reached components. Drift outside the computed radius means a missing edge to add. Edges that fired with no observable drift can be deprioritized in future risk rankings.

The Friday planner edit from the opening becomes a five-minute exercise: the graph fires three meaningful edges, the rule returns three evals, and the release note shows exactly which components were considered and which were checked. Agent pipeline testing stops being a choice between hours of reruns and shipping on hope. Prompt changes finally get what code changes got decades ago, a computed blast radius and a release gate sized to match it.