Legal AI RFPs Now Ask Who Controls the Architecture

Eighteen months ago, a general counsel evaluating legal AI vendors wanted to see benchmark scores. Today, the first question in an AmLaw 200 RFP is more likely to be: "Show me exactly what happens to our documents after we hit send." That single shift — from model quality to data architecture — is the most consequential change in legal AI procurement since ChatGPT entered the profession's vocabulary.

The trigger wasn't a breach. It was a disclosure. As firms built out agentic workflows across OpenAI, Anthropic, and other model providers in 2026, procurement and risk teams discovered that "we use a large language model" is not a single fact pattern — it's a spectrum of contractual retention terms, some measured in seconds and some in weeks. According to the Data Sovereignty section of the latest agentic_ai_infra industry report, the share of AmLaw 200 RFPs that open with a data-retention or architecture question — rather than a model-performance question — has moved from roughly 20% in early 2025 to an estimated 65%+ heading into Q4 2026. That reversal is the real story this quarter, and it has little to do with which model writes the better brief.

The Retention Divide That Started the Reckoning

The proximate cause is well documented and worth stating plainly, because most managing partners have heard about it secondhand rather than seen the terms side by side.

• OpenAI offers Zero Data Retention (ZDR) on eligible API endpoints for approved enterprise customers. Under ZDR, prompts and completions are not stored after the response is returned — no 30-day abuse-monitoring window, no residual copy sitting in a logging pipeline.
• Anthropic's standard commercial API terms retain inputs and outputs for up to 30 days for trust-and-safety review, even though the company commits contractually not to train on that data by default. True zero- or reduced-retention terms exist, but only through a custom enterprise agreement negotiated directly — not the default state.
• Microsoft's Azure OpenAI Service sits in between: enterprise deployments are generally excluded from training, but abuse-monitoring retention applies unless a firm successfully applies for a Limited Access / modified abuse-monitoring exemption.

None of these terms are secret. They're published in each provider's data usage and trust documentation. What changed is that law firm risk committees started reading them with the same scrutiny they apply to a custodian agreement, and found that the answer to "does our data leave the building" depends entirely on which API endpoint, which contract tier, and which product surface a given tool happens to be built on.

The table looks like a procurement footnote. It is actually the reason the entire buying conversation has changed.

Why "Which Model Is Smartest" Was Always the Wrong Question

For two years, legal AI marketing optimized for a single metric: benchmark performance on legal reasoning, contract review, or deposition summarization. That competition produced genuinely strong products — Harvey, CoCounsel, Lexis+ Protege, and Legora have all shipped credible gains on narrow tasks, and Claude's Cowork agent has pushed the ceiling on multi-step legal workflows. But benchmark leadership is a moving target that resets every few months as providers leapfrog each other, while a retention clause is a durable, auditable fact that a managing partner has to defend to a client's general counsel or a regulator.

Here's the uncomfortable math most RFPs miss: if a firm selects a legal AI platform because it is contractually and technically bound to a single model provider's API, the firm has not chosen a model — it has chosen that provider's retention terms, pricing trajectory, and roadmap, indefinitely, for every matter that touches the tool. A platform hard-wired to Anthropic's standard terms inherits the 30-day window whether or not the matter warrants it. A platform hard-wired to a single vendor's hosted infrastructure inherits whatever that vendor negotiates next year, with no seat at the table.

The honest framing isn't "vendors that touch a model are dangerous and vendors that don't are safe." Every serious legal AI platform, including RAGbase, uses frontier LLMs — none of them train their own GPT-4-class model from scratch. The real variable is who controls the architecture around the model: the retrieval layer, the connectors, the permissioning, the audit trail, and the contractual relationship with the model provider itself. That's the distinction buyers are now paying law firm IT and outside counsel to investigate before signing.

What Actually Leaves the Firm's Infrastructure

This is the detail that gets lost in vendor marketing on both sides of the debate, and it's worth being precise about it rather than making a blanket claim.

In a properly architected retrieval-based system, there are two categories of data, and they do not travel together:

1. What stays on firm infrastructure, permanently: the full document corpus (contracts, pleadings, discovery, correspondence), the vector index built from that corpus, the connector layer to DMS and email systems, matter-level and ethical-wall permissions, and the complete audit log of every query, retrieval, and output.
2. What is transmitted to the selected model provider, per query: a small set of retrieved chunks — typically a handful of passages totaling a few hundred to roughly two thousand tokens — plus the user's prompt, sent under whatever API terms the firm has itself selected and contracted for.

That second category is the only thing that ever crosses the firm's perimeter, and it's minimized by design: the retrieval layer selects the narrowest set of passages likely to answer the question, not the underlying documents themselves. A 400-page merger agreement never transits to the model provider; the three clauses relevant to the associate's question do, stripped of surrounding context the model doesn't need.

This is the architectural case for private AI deployment: the firm decides whether those retrieved chunks go to OpenAI under ZDR, to an Anthropic enterprise agreement with negotiated retention, or to a self-hosted open-weight model that never leaves the firm's own VPC at all. The retrieval, the permissions, and the logs don't change based on that choice — only the final, minimized transmission does. That's a materially different risk posture than a SaaS platform where the firm's only lever is trusting the vendor's existing contract with whichever model it happens to be built on.

The Architecture Comparison Buyers Should Actually Run

Procurement teams comparing legal AI vendors this quarter are increasingly running a four-category test rather than a feature checklist. Here's a representative version of how the categories stack up, generalized across current market offerings:

The pattern that matters for a managing partner isn't which column "wins" — it's that only one column gives the firm a lever to pull when a model provider changes its terms, raises prices, or gets acquired. That's the lever clients are now asking about directly, particularly in regulated industries and cross-border matters where data residency isn't optional.

The RFP Checklist This Quarter

Firms running AI RFPs in Q4 2026 are adding a short list of architecture questions ahead of any demo. The ones that consistently surface friction — or clarity — are:

• Where does the full document corpus reside at rest, and under whose contract?
• What exactly transits to the model provider per query — the full document, a summary, or a minimized retrieved chunk?
• Can we select, or later swap, the underlying model provider without re-architecting our workflows, connectors, or permissions?
• Who owns the audit log — can outside counsel or a client's security team review it directly, or does it live behind the vendor's support portal?
• Are matter-level permissions and ethical walls enforced at the firm's identity layer, or reconstructed inside the vendor's own access model?
• If the model provider changes its retention policy next year, does the firm's exposure change automatically, or is it insulated by its own negotiated terms?

Firms that can't get clear answers to the second and sixth questions in particular are, in effect, delegating their data governance policy to a vendor's product roadmap. That's a defensible position for low-sensitivity workflows like public case law research — a workload well suited to tools like case search — and a much harder position to defend for privileged work product, M&A diligence, or regulated-industry matters.

Model-Agnostic Is Becoming the Default Procurement Requirement

The next phase of this shift is predictable because it already happened in cloud infrastructure a decade ago: large enterprises moved from single-cloud bets to deliberate multi-cloud strategies specifically to avoid vendor and pricing lock-in. Legal AI is following the same curve, compressed into a couple of years instead of a couple of decades.

Expect three things to become standard RFP language by mid-2027:

• Explicit model-agnosticism requirements — firms specifying that any platform must support at least two frontier model providers plus a self-hosted option, so retention terms become a firm-level choice rather than an inherited constraint.
• Workload-based model routing — using a fast, cheaper model with aggressive retention terms for low-sensitivity research, and a zero-retention or air-gapped configuration for privileged drafting, within the same platform.
• Retention terms as a negotiating lever, not a fixed cost of doing business — firms using the OpenAI/Anthropic divide itself as leverage to negotiate better terms with whichever provider wants the business, something only possible if the firm's architecture isn't welded to one vendor's existing contract.

The firms treating this as a one-time RFP question are underestimating it. Model providers' retention and training policies have changed multiple times in the past two years alone, and will keep changing as regulatory pressure (EU AI Act implementation, state-level AI statutes, client-mandated security questionnaires) increases. An architecture that lets a firm absorb that change at the contract level, without touching its workflows, connectors, or logs, is a durable advantage. An architecture that requires a platform migration every time a model provider updates its terms is a recurring cost that compounds with every matter the firm has run through it.

For managing partners and CIOs reviewing RFPs this quarter, the useful exercise isn't ranking model benchmarks — it's mapping exactly what leaves your infrastructure, what stays, and who gets to change that answer next year. Our AI for law firms guide walks through that mapping in more detail, including how a retrieval-first, model-agnostic architecture lets a firm keep its documents, permissions, and audit trail under its own control while still choosing — and changing — which frontier model actually answers the question.