Retrieval-augmented generation and structured-data agents are often presented as competing architectures. Usually they are answers to different questions.
RAG is well suited to finding and synthesizing meaning from unstructured material: policies, manuals, research, meeting notes, product documentation, and other text-heavy sources.
A structured-data agent is better suited to precise questions about entities, states, filters, and aggregations: open invoices, orders by region, inventory below a threshold, or the current status of a specific account.
The right choice follows from the shape of the information and the kind of correctness the user expects.
Ask what makes an answer correct
Consider two questions:
- “What does our return policy say about damaged products?”
- “How many damaged-product returns were approved last quarter?”
The first requires interpretation of policy text. The second requires a defined metric, a date boundary, and exact aggregation over operational records.
A vector search over policy documents can support the first. It should not be trusted to calculate the second. A database query can answer the second, but it may not explain the policy language behind it.
Before choosing an architecture, define the correctness condition:
- Is a relevant passage and faithful synthesis enough?
- Must the answer be complete across all matching records?
- Does the user need an exact number?
- Does “current” mean seconds, hours, or the last document index?
- Must every answer map to a specific source record or document version?
Those questions expose whether the task is retrieval, computation, or both.
What RAG is good at
A RAG pipeline typically divides source material into retrievable units, represents those units for search, selects relevant context for a question, and asks a model to answer from that context.
Its strength is semantic access. Users do not need to know the exact title, keyword, or location of a passage. They can ask in natural language and receive a synthesized answer across relevant text.
RAG works especially well when:
- the source is primarily unstructured text;
- wording varies between the question and the source;
- the task is explanation, comparison, or synthesis;
- citations to passages are useful; and
- approximate retrieval can be followed by grounded generation.
Its central failure mode is incomplete evidence. Retrieval can miss a relevant passage, return a stale version, or provide context that sounds sufficient but is not comprehensive.
What structured tools are good at
Structured systems make entities, fields, relationships, and constraints explicit. An agent can access them through narrowly defined tools or an intermediate semantic layer.
Their strength is deterministic selection and calculation. If the schema and metric definition are correct, a query can identify every matching record and compute an exact result.
Structured access is the better foundation when:
- the question contains filters, joins, counts, sums, or time windows;
- the answer must reflect current operational state;
- completeness matters;
- permissions differ by row, field, tenant, or account; or
- the system may take action on a specific record.
The failure modes are different from RAG. A query may use the wrong field, join at the wrong grain, misread a business definition, or produce a technically valid number with the wrong meaning.
That is why free-form text-to-SQL is rarely the whole solution. Safer systems expose governed views, approved metrics, typed tools, schema descriptions, and validation around the generated operation.
Do not put structured facts into a document-shaped disguise
A common shortcut is to convert every database row into text, embed it, and use vector search for all questions.
This may support fuzzy discovery, such as finding accounts that mention a particular issue. It is a poor default for exact operational analysis.
Vector similarity does not guarantee exhaustive retrieval. The top results are not necessarily all results. A generated summary cannot recover records the retriever never returned, and adding more chunks does not create a reliable aggregation engine.
If the underlying question is “Which records satisfy these conditions?”, preserve the structured representation.
Hybrid systems need a clear boundary
Many useful assistants need both modes.
Imagine an operations assistant asked:
Which enterprise accounts have unresolved incidents, and what does the escalation policy require us to do next?
The first clause belongs to structured operational data. The second belongs to policy documents. A hybrid system can:
- call a governed tool to retrieve the relevant accounts and incidents;
- retrieve the current escalation-policy sections;
- join the evidence at the application layer; and
- generate a response that keeps operational facts and policy interpretation distinguishable.
The important design choice is not simply to use both. It is to retain provenance for each claim. The user should be able to tell which statements came from database records, which came from documents, and which are model-generated interpretation.
Route by intent and required evidence
A practical router does not need to be elaborate. It can classify a request along a few dimensions:
- source shape: document, structured record, or mixed;
- operation: retrieve, compare, calculate, explain, or act;
- freshness: indexed knowledge or live state;
- precision: approximate relevance or exhaustive correctness; and
- authority: read-only answer or external change.
That classification can select a document retriever, a typed data tool, or a planned sequence that uses both.
For ambiguous requests, the system can ask a clarifying question or return separate interpretations instead of pretending the distinction does not matter.
Evaluate each path differently
Document retrieval should be evaluated for relevance, coverage, faithfulness, citation quality, and version handling.
Structured access should be evaluated for filter correctness, join grain, aggregation logic, metric definitions, permissions, and reconciliation against known results.
Hybrid answers need an additional test: whether claims remain correctly attached to their source type as information moves through the system.
A single “answer quality” score hides these differences. Architecture-specific evaluation makes failures actionable.
The decision rule
Use RAG when the task is to find and interpret meaning in unstructured sources. Use structured tools when the task requires exact records, current state, deterministic computation, or controlled action. Use both when the user’s question genuinely crosses that boundary, and keep the evidence paths separate even when the final answer feels unified.
The goal is not to make every source look the same to a model. It is to preserve what makes each source trustworthy.
