Production LLM Application Security Beyond Prompt Injection

Most teams learned prompt injection as a parlor trick: slip the model a sentence that says "ignore previous instructions," watch it comply, file a bug. That mental model is now actively dangerous. When an LLM reads email, retrieves documents, and calls APIs on your behalf, the injection stops being a content problem and becomes a control-flow problem. LLM application security cannot be solved at the input layer, because the input layer has no hardware-enforced boundary between trusted instructions and the untrusted data the model is paid to ingest.

You know the mechanics. The question now is architectural: what to do about the four vulnerability classes that have started breaking production systems: RAG retrieval manipulation, memory persistence exploits, tool poisoning, and cross-session data leakage. Each one maps to a specific architectural countermeasure, and together they form a defense-in-depth stack you can audit against today.

Why Input Filtering Cannot Be Your Perimeter

The token stream is the root cause. At inference, a language model consumes a single undifferentiated sequence. Your system prompt, a retrieved document, a tool output, and a just-fetched webpage all collapse into one context window with no hardware-enforced boundary between trusted instructions and untrusted data. Input filtering tries to intercept malicious text before it enters that stream, but you are scanning adversarial prose for adversarial intent with no structural advantage. The attacker has infinite freedom in phrasing. The defender has to win every time.

ForcedLeak, disclosed by researcher Noma in September 2025 as a CVSS 9.4 vulnerability in Salesforce Agentforce, illustrates the pattern. An attacker hid instructions in a Web-to-Lead form that sat dormant in the CRM until an employee's AI agent processed the lead, executed the hidden payload, and exfiltrated records to an expired allowlist domain re-registered for roughly five dollars. Every request was ordinary traffic to an approved destination.

The OWASP Top 10 for LLMs ranks prompt injection as the top risk, yet most teams build controls only for direct injection. Indirect injection research demonstrated years ago that adversarial text in external content can coerce models into exfiltrating data such as SSH keys, succeeding in a high percentage of trials across multiple experiments. The agent was doing exactly what it was designed to do with attacker-written content.

The shift is to stop detecting injection and start constraining action. The rest of this article maps the four vulnerability classes now breaking production systems (RAG retrieval manipulation, memory persistence exploits, tool poisoning, and cross-session data leakage) to the architectural gate each one demands.

The Vulnerability Classes That Actually Break Production

The MITRE ATLAS framework catalogs adversarial techniques against AI systems the way MITRE ATT&CK maps traditional enterprise threats. Borrowing that lens, four vulnerability classes have matured from research demos into production incidents. Each targets a different component of the agent stack, and each requires a different countermeasure. Treating them as one undifferentiated problem called "prompt injection" is why most teams underinvest in the architectural fixes that actually work.

Vulnerability Class	Target Component	Core Countermeasure
RAG retrieval manipulation	External knowledge corpus	Source provenance tagging and ingestion gating
Memory persistence exploits	Long-term agent memory store	Provenance-tagged, TTL-bounded memory partitions
Tool poisoning	Function execution boundary	Deterministic, version-checked capability contracts
Cross-session leakage	Multi-tenant context window	Namespace-enforced retrieval assertions

RAG Retrieval Manipulation and Data Poisoning

Retrieval-augmented generation pipelines are attractive because they let the model cite fresh, proprietary data. They are equally attractive to attackers, because the retrieval layer inherently trusts unstructured external content more than it trusts the model's trained weights. research on RAG poisoning attacks demonstrates that an attacker who can inject even a small number of crafted documents into an indexed corpus can bias retrieval toward malicious passages, skew model outputs, and in some designs leak private indexed data through crafted queries.

The structural flaw is that the retriever does not distinguish "a document the company wrote" from "a document an attacker uploaded." Knowledge base ingestion is treated as a content pipeline, not a security boundary. Three countermeasures materially reduce exposure:

Source Provenance Tagging

Stamp every chunk with its origin, classification, and trust level at ingestion time, then propagate that metadata into the prompt so the model can weight sources differently.

Retrieval-Time Sanitization

Quarantine chunks whose instruction-like phrasing or content density deviates from the corpus norm before they reach the context window.

Human-Gated Ingestion for External Sources

Anything pulled from the open web, customer uploads, or third-party feeds enters a review queue, not the live index.

The defense is not "make the model ignore bad documents." It is "stop routing untrusted text into the trusted context stream without provenance."

Memory Persistence Exploits in Autonomous Agents

Every other vulnerability in this article describes an injection that lives for one request. Memory is the only class that turns a transient event into a permanent compromise. A payload written to long-term memory outlives the session where it was delivered, the model version that first processed it, and the input filter that missed it. The injection travels through time, which is why the OWASP agent security guidance treats agent memory as a first-class attack surface.

The attack chain is short. An attacker delivers an injection through any channel the agent ingests: an email body, a meeting transcript, a summarization task. The agent processes the input, writes a condensed summary into its vector store, and strips original context while preserving the payload. Weeks later, a different session retrieves that memory as trusted internal state and executes it. No filter re-scans the read, because the system treats persisted memory the same way it treats its own scratchpad. The injection has been laundered through the trust boundary.

The fix is provenance-tagged, expiry-bounded memory. Every write stamps its origin channel and trust level, just as RAG ingestion stamps document provenance. Reads respect those tags: instructions originating from user-supplied content get quarantined, not executed. For anything from an untrusted channel, apply TTL-based expiry so injected payloads age out rather than persisting indefinitely. High-sensitivity operational instructions should not share a partition with user-supplied notes. Segregation limits blast radius when one partition falls, and periodic auditing catches payloads that slipped past filters whose detection has since improved.

Tool Poisoning and Function Execution Hijacking

This is where prompt injection crosses into remote code execution territory. When an agent can call tools, the model's output directly drives real actions: database queries, file writes, API calls, money movement. Analysis of tool-calling attacks demonstrates that without strict, deterministic permission boundaries, a compromised orchestrator can chain tool calls to escalate privileges and execute attacker-chosen operations, up to and including arbitrary code execution in poorly isolated runtimes.

The failure mode is not exotic. An agent reads a poisoned document. The document instructs it to call a file-write tool with attacker-controlled arguments. The agent complies, because the instruction arrived through a channel indistinguishable from a legitimate user request. Native model-side permission checks, when they exist, are themselves model outputs and therefore themselves injectable.

The architectural fix mirrors how distributed systems handle untrusted code:

Typed Tool Calls, Never Free Text

Consequential actions cross the boundary as typed tool calls, never free text. The contract inspects approve_invoice(vendor, amount) cleanly because the action is already structured. Parsing prose back into a schema reintroduces a model into the validation path.

Deterministic Gates, Not LLM Self-Checks

A deterministic contract is a declarative file the orchestrator loads at startup and evaluates before every tool call. It checks dollar limits, allowed destinations, and rate caps before any API call fires, and it is not a prompt; it is code. The second-LLM-as-judge pattern reintroduces the exact vulnerability one layer down. Here is what one looks like:

# agent_contract.yaml
agent_id: accounts-payable-bot
role: invoice_processor
policy:
  approve_invoice:
    max_amount_usd: 50000
    allowed_vendors:
      ref: vendor_registry.approved
    require_human_above_usd: 10000

At runtime, an injected action proposing approve_invoice for $1,200,000 hits the $50,000 ceiling. The action is discarded, a human reviewer is notified, and no API call fires. An injected instruction arriving at 2am never matters, because the deterministic gate evaluates the action, not the intent behind it.

Transient, Action-Scoped Credentials

The agent starts each session with no dangerous permissions. It requests narrow elevation per action, the gate approves or denies, and the credential evaporates the instant the action completes.

Cross-Session Context Window Leakage

Context window leakage is the vulnerability your existing data-loss prevention stack is blind to. Traditional shared-state bugs live in a database row or a cache entry you can audit, roll back, or quarantine after detection. A context window leak happens in the model's working memory during inference. By the time the response is generated, the contamination has already influenced the output, and no log entry, no row-level access control, and no DLP alert captures what bled across. Cross-session leakage research documents how residual context, shared prompt caches, and reused orchestrator state have carried information between sessions that should have been separated. Guidance on GenAI tenant isolation makes the structural point: in shared compute, the boundary between one tenant's data and another's query must be enforced, not hoped for at the provider level.

The two surfaces teams miss most are prompt-cache contamination and process-state reuse. A cached prefix from tenant A served to tenant B is silent, fast, and invisible to application logging. An orchestrator that reuses intermediate reasoning across requests passes context forward without any explicit write. Each exploits a gap traditional tooling cannot see:

Leakage Surface	What DLP Tooling Misses	Isolation Countermeasure
Provider prompt cache	Cache hits are opaque at the application layer; no query log records the prefix served	Cache-busting or explicit opt-out for multi-tenant traffic
Orchestrator process state	Intermediate reasoning is never persisted, so no audit trail exists	Fresh process state per request for sensitive workloads
Shared vector store	Cross-tenant retrieval returns valid-looking chunks with no tamper signal	Namespace-enforced partitioning with retrieval-time tenant assertions

The concrete fix is namespace-enforced retrieval assertions as a runtime invariant, not a deployment-time configuration. Every retrieval call must prove tenant ownership of the returned chunks at query time. If the assertion fails, retrieval returns nothing. This belongs in the code path, not in infrastructure settings a misconfiguration can silently disable.

The Defense-in-Depth Stack for LLM Application Security

The four vulnerability classes share one design rule: deterministic code validates every proposed action before it executes. The defense-in-depth architecture for GenAI is not a checklist of independent controls. The gates form a pipeline where each assumes the previous one already failed.

The NIST AI RMF supplies the governance vocabulary. Three engineering commitments hold the pipeline together, and their composition is where teams get it wrong.

First, action-scoped least privilege. Privilege attaches to the action, not the agent. Elevation is transient and per-request, whether the action is a RAG retrieval, a memory write, or a tool call.

Second, zero-trust machine identities. Every internal call between agent components is authenticated as if it came from an untrusted actor, because functionally it may have. Zero-trust patterns for ML systems formalize this shift. Composition matters: a capability contract authenticates against a machine identity, and that identity is only as trustworthy as the memory store that provisioned it.

Third, capability contracts at every boundary. Each consequential action crosses a version-controlled, deterministic check that lives outside the model. The model proposes. Code disposes.

If you can build only one gate first, build the tool gate. Tool execution has the largest blast radius (irreversible actions like money movement and data deletion) and the clearest contract boundary. RAG provenance and memory tagging reduce information leakage. Tool contracts stop irreversible damage. Prioritize by what you cannot undo.

One composition failure mode deserves attention. A capability contract that authenticates against a machine identity compromised through a memory persistence exploit will rubber-stamp attacker actions with full legitimacy. The gate works. The identity it trusts is poisoned. Layering controls does not save you when trust flows downstream from a corrupted source.

Audit Your Stack: Where to Start Monday

You have the four gates. The reason most teams have not built them is not technical ignorance. It is normalized deviance. You connect an agent to a production database, and nothing bad happens. You wire tool calling into a payment API, and nothing bad happens. Each uneventful day makes the next risky connection feel safer. The absence of incidents becomes evidence that the controls are fine, when really it is evidence that nobody has tried yet.

Normalized deviance mirrors the pattern safety researchers have long documented in aerospace and nuclear incidents: a streak of operations outside the documented safety envelope, each one surviving, each one quietly lowering the bar for the next. Years of warnings about prompt injection have not moved spending on action-layer controls because the deviance curve keeps reassuring everyone that this is fine.

The antidote is to act before the streak breaks. Start Monday with this sequence:

1. Inventory actions by blast radius. List every action your agents can take: database queries, file writes, API calls, tool invocations. Sort by the worst outcome if each fires when it should not.
2. Write deterministic contracts for the high-risk ones. Dollar limits, allowed destinations, rate caps, human-review thresholds. Put them in version control, outside the model.
3. Audit ingestion paths. RAG corpus, agent memory, tool outputs, web fetches. Which carry content from sources you do not control? Tag provenance at ingestion.
4. Verify tenant isolation. If you serve multiple users or organizations, confirm context isolation, vector store namespacing, and prompt cache behavior.
5. Only then, harden inputs. Instruction hierarchies, classifier-based detection, and system-prompt hardening still belong in the stack. They just cannot be the only thing in it.

The teams that weather the next wave of agent incidents will not be the ones with the cleverest filters. They will be the teams that treated compromise as the baseline assumption and shipped enforcement controls anyway, the ones who drew the line at irreversible actions and refused to let an agent cross it.