Legal AI's Reference Architecture Has Standardized. Who Owns It?

In September 2025, most firms evaluating legal AI were asking "which model is best?" One year later, that question has become almost irrelevant. Across agentic AI infrastructure briefings this quarter, a consistent pattern has emerged: the architecture underneath legal AI tools has converged into a standard, three-layer stack — a gateway that governs how data reaches model providers, a retrieval layer that indexes firm knowledge with permissions enforced, and an evaluation layer that scores outputs against golden datasets. The model itself has been demoted to a swappable part.

This is not a minor technical footnote. It's the single most important development in legal AI infrastructure this year, because it reframes the entire buying decision. The question is no longer "which vendor has the best model." It's "who owns the gateway, the retrieval layer, and the eval stack — and what happens to my firm's leverage when I don't?"

The Stack That Won: Gateway, Retrieval, Eval

The convergence didn't happen by accident. It happened because the alternatives kept failing in predictable, expensive ways.

Early legal AI deployments in 2023-2024 were largely single-model wrappers: a chat interface bolted onto GPT-4 or Claude, with light prompt engineering standing in for real retrieval. These systems hallucinated on citation-heavy work, had no audit trail a general counsel could defend in a malpractice review, and broke every time a model provider changed an API or deprecated a version. Firms learned the hard way that a legal AI product is not a chatbot with a law firm logo — it's an infrastructure problem with three distinct, separable concerns:

• Gateway layer — governs which model receives which data, under what contractual terms (zero-data-retention, no training-on-inputs, regional data residency), and logs every call for audit purposes.
• Retrieval layer — indexes the firm's actual corpus (matters, precedent, contracts, internal memos) with permissions enforced at the document and clause level, so an associate's query never surfaces a document she isn't cleared to see.
• Eval layer — continuously scores agent outputs against golden datasets (known-correct answers on real firm work product) so quality regressions get caught before they reach a partner's desk, not after a client complaint.

The model sits underneath all three, swappable by design. When a firm wants to move from one frontier model to another — because pricing changed, because a new model tests better on the firm's own eval set, because a client mandates a specific provider — that swap should take an afternoon of configuration, not a re-platforming project. That's the architectural insight the market has now standardized around, and it's exactly the structure RAGbase Legal built its private AI deployment around well before it became the consensus reference design.

The Part Everyone Gets Wrong: What Actually Leaves the Building

The honest technical story here is more nuanced than "we keep everything in-house and they don't." Every legitimate agentic legal AI system, RAGbase Legal included, ultimately calls out to a frontier LLM provider for reasoning and generation. Claiming otherwise would be dishonest and easily disproven by any technical due diligence process a firm's IT security team runs.

The real distinction is architectural, not binary:

• What stays on the firm's infrastructure: the full document corpus, the vector index built from it, the permissions graph mapping who can see what, the complete audit logs of every query and retrieval, and the agent orchestration logic that decides which tools to call and in what sequence.
• What leaves, selectively: only the specific retrieved chunks relevant to a given query — typically a few paragraphs or clauses, not entire documents — sent to the model provider the firm has chosen, under the firm's own negotiated API terms, including zero-data-retention where required.

That distinction matters enormously in practice. A firm running a fully in-house build still sends chunks to OpenAI, Anthropic, or Google for inference — there's no way around that unless the firm is running open-weight models on its own GPUs, which almost no AmLaw 200 firm does today for cost and performance reasons. The difference is who controls the gateway deciding what gets sent, who owns the retrieval index deciding what's retrievable in the first place, and who holds the logs proving what happened for every single query, forever.

With a rented, per-seat SaaS product, the firm typically has none of that visibility. The retrieval layer is the vendor's black box. The eval methodology is the vendor's internal QA process, disclosed in a sales deck, not an auditable log. The firm is trusting the vendor's architecture diagram rather than owning one.

Build vs. Rent vs. Own: A Structural Comparison

Most firms frame the decision as "build it ourselves" versus "buy a SaaS seat license." That framing misses the third option that the standardized reference architecture actually enables: a productized private deployment the firm owns and operates, without the multi-year engineering build.

The SaaS column is where most of the market currently sits. Harvey, CoCounsel, Lexis+ Protégé, and Legora are all, at their core, sophisticated implementations of the same gateway-retrieval-eval architecture — sold as a managed service where the firm rents access rather than owning the layers. That's a legitimate model for firms that want zero infrastructure responsibility and are comfortable with per-seat pricing that, based on publicly disclosed enterprise agreements, commonly runs $100-400+ per user per month depending on tier and usage caps. For a 400-attorney AmLaw 200 firm, that's a $500K-$2M annual run rate for a black-box retrieval and eval layer the firm can never fully audit or take with it.

Claude Cowork and similar consumer-grade agentic assistants sit in a different bucket entirely — general-purpose agents with real capability but no legal-specific retrieval index, no clause-level permissions model, and no eval stack built against legal-specific golden datasets. They're useful for general document work; they are not a substitute for a firm-grade retrieval and evaluation layer built on privileged legal corpora.

Why the Middle Layer Is Where the Value (and the Risk) Actually Sits

The model is not where legal AI differentiation lives anymore, and treating it that way is a strategic mistake. Frontier model quality has converged enough — across GPT-5-class, Claude Opus-class, and Gemini-class models — that benchmark gaps between top providers on legal reasoning tasks have narrowed to low single digits on most public evaluations. What hasn't converged, and what actually determines whether a legal AI deployment is safe to put in front of a client, is:

• Retrieval precision — does the system pull the actual controlling precedent from the firm's own matter history, or a generic web-trained approximation of it?
• Permission fidelity — does an associate's query respect ethical walls and matter-level access controls with zero leakage, every time, not just in the demo?
• Eval rigor — is every model version and every prompt change tested against a golden dataset of the firm's own historically correct answers before it touches a live matter?

These three layers are exactly where a firm's competitive and risk position is made or lost — and they're precisely the layers a rented SaaS seat keeps opaque. A firm that owns its retrieval index and eval stack can prove, to a GC, to a malpractice carrier, to a court in a privilege dispute, exactly what data the system touched and why it produced a given answer. A firm renting a black box can only point to a vendor's marketing claims.

This is also why model swapping matters more than most firms initially assume. When a new frontier model tests measurably better on a firm's own golden dataset — not a vendor benchmark, the firm's actual historical work product — the firm should be able to route to it in an afternoon. In a rented architecture, that decision belongs to the vendor's roadmap, not the firm's.

The Eval Stack: The Layer Nobody Priced In Two Years Ago

If 2023-2024 legal AI conversations centered on prompt engineering and 2025 conversations centered on retrieval-augmented generation, 2026's conversations are centered on evaluation — and for good reason. An eval stack against golden datasets is the only mechanism that turns "the AI seems pretty good" into a defensible, auditable quality process.

Concretely, a mature legal AI eval layer does three things a firm's risk committee should be asking about directly:

1. Regression testing on every model or prompt update — so a provider's silent model update doesn't quietly degrade contract review accuracy without anyone noticing until a client flags it.
2. Golden dataset scoring built from the firm's own matters — not generic legal benchmarks, but the firm's historically verified correct answers on its own document types, jurisdictions, and practice areas.
3. Per-workflow accuracy tracking — due diligence summarization, deposition prep, clause extraction, and litigation research each need separate eval tracks, because a system that's 94% accurate on contract summarization can be materially worse on multi-jurisdictional research synthesis.

Firms that skip this layer are, in effect, running production legal work on an untested pipeline every time an underlying model updates. That's not a hypothetical risk — it's the exact failure mode that has generated the malpractice and confidentiality incidents referenced across legal AI adoption surveys this year, where firms discovered accuracy drift only after a partner caught a fabricated citation in a filed brief.

What This Means for the Buying Decision

For managing partners and CIOs evaluating agentic legal AI heading into 2027 budget cycles, the standardized reference architecture actually simplifies the decision, once you stop evaluating vendors on model quality alone. The right questions are:

• Do we own the retrieval index, or are we renting access to someone else's?
• Can we see and audit the eval methodology, or are we trusting a vendor's internal QA?
• If we want to swap models next year, is that a configuration change or a migration project?
• Does our permissions layer map to our actual DMS and ethical wall structure, or to the vendor's generic org chart?
• What contractual terms govern the data that does leave our infrastructure, and did we negotiate them or inherit them?

Firms that can't answer these today are not necessarily behind — most of the market can't answer them either. But the gap between firms that own this architecture and firms that rent it opaquely is going to widen, not narrow, as eval sophistication and model-swapping cadence become genuine competitive differentiators rather than IT trivia. Our AI for law firms guide walks through how to map your current corpus, permissions structure, and workflows against this reference architecture before committing to either a build or a rental.

The reference architecture question has been answered. What remains open is who holds the keys to each layer of it inside your firm — and whether that answer is something your general counsel, your malpractice carrier, and your largest client can actually audit when they ask.