Natural Language Querying for Trusted Answers

Automated systems achieved 65.45% execution accuracy, compared with 92.96% for human experts, in a 2024 examination of the BIRD database-querying benchmark described in the text-to-SQL survey. That's the enterprise challenge in miniature: a system can understand the wording, generate executable SQL, and still answer the wrong business question. For CIOs and data leaders, natural language querying is therefore a question of controlled access to business meaning, not just a more convenient chat interface.

Table of Contents

What Natural Language Querying Means for Enterprise Data

Natural language querying translates an everyday question, typed or spoken, into an executable database query. A trustworthy implementation then checks the result and explains its basis, rather than returning a confident paragraph unsupported by database evidence. The database-specific practice is commonly called text-to-SQL or a natural-language interface to databases, abbreviated NLIDB.

Say a finance director asks for revenue by region after the latest close. They know the business question, but probably not which billing objects feed the warehouse, how credit notes are represented, or which customer hierarchy defines a region. The analyst who knows those details might not be in the meeting.

Natural language querying aims to remove that dependency for established questions. Users shouldn't need to learn table names, join conditions, filters, or SQL syntax just to retrieve an approved measure.

More than conversational access

This isn't a newly discovered database problem. The VLDB paper on natural-language database interfaces describes a research history extending across decades, including intermediate logical representations proposed during the 1980s. Modern language models change how teams implement the interface, but they don't remove the need to map words to database structures.

In an enterprise, that mapping includes business decisions. Does revenue mean invoiced revenue or recognized revenue? Does region follow the selling organization, customer address, or delivery destination? Should the director see every legal entity?

Think of the interface as a bilingual analyst who understands both business language and database structures, but won't answer until the official definition is clear.

The distinction matters because a database schema describes how information is stored, not necessarily how leaders have agreed to measure performance. A customer identifier can be technically valid while referring to the wrong customer grain.

For enterprise leaders, the useful outcome is governed self-service: routine questions answered against approved meanings, with specialist involvement when interpretation is required. Ease of asking is valuable only when the organization can defend the answer.

How a Question Becomes a Safe Query

A reliable implementation separates interpretation from execution. For a request such as revenue last quarter by region, the system needs to establish meaning before it decides how to retrieve rows.

The translation sequence

  1. Parse the request. Identify the metric, time period, business entity, filters, and requested comparison. Revenue is the measure, last quarter supplies a time constraint, and region supplies the grouping.
  2. Ground the words. Resolve the measure against the business glossary and use the approved fiscal calendar. If competing definitions remain, ask the user to choose.
  3. Plan the query. Establish the grain, relationships, and aggregation order. Joining invoice lines to multiple customer addresses before summing can multiply revenue even when every SQL expression is valid.
  4. Generate platform-specific SQL. Render the approved plan in the dialect used by the target platform, such as SQL Server or Databricks SQL. Date functions and supported syntax must match that environment.
  5. Validate and execute. Check referenced objects, allowed operations, and the execution plan, then run under the user's effective permissions. Return the result alongside the relevant tables, filters, aggregation, and freshness.

An intermediate representation can help separate business interpretation from SQL construction. The NatSQL paper reports that adding NatSQL to RAT-SQL with GAP produced a 2.2-percentage-point absolute improvement over the strongest prior system on Spider by simplifying SQL-specific complexity before conversion to executable SQL.

That finding supports an architectural principle, not a promise about enterprise accuracy: make the intended calculation inspectable before rendering the final statement.

What a question must resolve before SQL is written

Question pattern Must be resolved first Safe query shape
What was revenue last quarter by region Certified revenue measure, fiscal quarter, region hierarchy Aggregate the approved measure by region for that fiscal period
Which plants missed on-time delivery this month On-time definition, plant master, shipment grain, calendar month Count late shipments against the approved promise date by plant
How many active customers do we have Active definition, customer grain, as-of date, deduplication rule Count distinct customers matching the certified active flag
Why did margin fall Margin formula, cost components, comparison period, allowed drivers Refuse a single guess; clarify formula, then compare certified margin components

Execution accuracy asks whether generated SQL returns the correct result. That's more useful than requiring identical SQL text, because differently written statements can be logically equivalent.

However, a correct-looking result isn't sufficient validation. A query might match the expected answer by coincidence, omit an authorization constraint, or fail when the underlying records change. Tests should therefore inspect the query's logic as well as its output.

The implementation boundary should be explicit: resolve meaning against a governed semantic model, then generate SQL. Don't give a conversational interface unrestricted access to raw operational tables and expect prompting to supply missing business rules.

Where Published Accuracy Falls Apart

Published results need their evaluation context attached. The following scorecard separates execution accuracy, exact-set matching, and task completion rather than treating them as interchangeable measures.

Natural language querying accuracy by benchmark

Evaluation What it stresses Reported result
Spider execution accuracy, May 2020 Cross-domain text-to-SQL 53.5% execution accuracy
Spider execution accuracy, March 2023 Later systems, same benchmark 85.3% execution accuracy
Spider exact-set match, comparable span Structural SQL matching rather than returned results 65.6% to 74.0%
Spider 1.0, GPT-4-class methods Academic cross-domain schemas 91.2% execution accuracy
BIRD, GPT-4-class methods Harder values and knowledge 73.0% execution accuracy
BIRD, 2024 automated vs human Same realistic items in a separate examination 65.45% automated, 92.96% human
Spider 2.0-lite, GPT-4-class Large schemas, dialects, workflows 5.7% execution accuracy
Spider 2.0 full tasks, o1-preview agent Iteration over real project context 21.3% of tasks solved
BIRD-Ent and Spider-Ent Large enterprise schemas and scattered knowledge 39.1% and 60.5% exact match

The historical Spider and separate BIRD comparison come from the survey cited in the introduction; the enterprise exact-match evaluations are documented in the enterprise benchmark paper linked under governance. The Spider 2.0 paper reports the remaining benchmark comparisons, including the 5.7% execution accuracy on Spider 2.0-lite and 21.3% full-task completion results.

The visual summarizes the earlier benchmark improvement. Its endpoint should be read as March 2023, not as a current production-accuracy estimate.

A chart showing an increase in Spider execution accuracy from 53.5 percent in May 2020 to 85.3 percent today.

Why enterprise questions are different

The difficult part isn't always writing a join. It's identifying which relationship is valid when a schema contains overlapping objects, historical mappings, and undocumented exceptions.

Enterprise tasks can require project documentation, sample-value inspection, several SQL dialects, and repeated investigation. A field named status might describe an order, shipment, cancellation workflow, or financial posting. Selecting a real column doesn't establish that it represents the user's intent.

Likewise, a query can apply the wrong predicate without producing an error. Filtering by invoice date instead of posting date can return a plausible financial total that doesn't reconcile to close.

Decision rule: A public benchmark indicates capability. It doesn't authorize an operational or executive decision.

Build acceptance tests from production-like questions, including cross-system requests and intentionally ambiguous wording. Pair those tests with data observability so reviewers can distinguish a query-generation failure from stale, incomplete, or unexpectedly changed source data.

The score to trust is the one earned against your definitions, schemas, and access boundaries.

Benefits That Depend on a Semantic Layer

A semantic layer connects business terms to approved measures, dimensions, relationships, and aggregation rules. Its value is practical: users can ask familiar questions without forcing the model to reconstruct business logic from column names.

The benefits are conditional. If the model invents a relationship or substitutes a convenient measure, faster access accelerates the delivery of unreliable numbers.

Less queueing for established questions

A plant manager asking for yesterday's throughput shouldn't need an analyst to rebuild a standard calculation. A finance partner requesting revenue by region after close should be able to use the same measure that appears in the approved financial dashboard.

When these questions run against certified definitions, analysts can spend more time investigating exceptions and designing new measures. They're no longer required to act as human query editors for every routine lookup.

  • Operations leaders get an accessible route to established performance measures.
  • Finance teams can reconcile conversational answers to existing reporting.
  • Analysts retain responsibility for novel analysis rather than repetitive retrieval.

Consistency across systems and reviews

For a data leader, the larger benefit is definition reuse. Revenue shouldn't acquire a different meaning because someone asks through a conversational interface instead of Power BI.

A unified data modeling approach helps establish shared entities and relationships across the analytics estate. Curated cross-system views can then connect finance, supply chain, and customer information before a user asks a question, rather than leaving the interface to improvise those relationships at runtime.

For example, a warehouse supervisor asking which aisle missed its pick target has a bounded retrieval task. Asking the same interface to recommend a new safety-stock policy requires assumptions about demand, service levels, lead times, and acceptable risk. Treating those requests as equivalent would confuse access to information with responsibility for a decision.

Auditability is another conditional benefit. A reviewer should be able to challenge the period, grouping, or measure without reconstructing an entire chat history. Present business-readable evidence alongside the result, while making technical query details available for inspection.

Public-sector and operations teams can gain a shorter path from question to action. CIOs and CDOs gain something different: a managed self-service channel that doesn't require another uncontrolled spreadsheet extract.

Those outcomes justify a pilot when the semantic foundations exist. Model fluency alone doesn't.

Questions Worth Asking in Real Operations

Operational questions expose assumptions that a polished demonstration can hide. Each scenario below has a useful question, a specific trap, and evidence that a reviewer should demand.

The operational checks can be summarized visually before applying them to individual decisions.

An infographic detailing important questions for operational data analysis, highlighting potential traps and requirements for trustworthy answers.

Delivery promises and financial explanations

Manufacturing scenario. A planner asks which distribution centers missed the on-time promise last week. Suppose the SAP data contains requested, confirmed, and revised promise dates: choosing whichever column sounds closest changes the result.

The answer should name the certified promise date, reporting week, late-shipment count, and extraction freshness. Retain the original request and generated query, and provide a route to the logistics owner before the planner uses the result for a consequential supplier or customer decision.

Finance scenario. A controller asks why margin moved after close. Retrieving a margin difference doesn't prove its cause.

The system should establish the approved margin formula and compare its recognized components, such as revenue and relevant costs. It should label unexplained movement as unresolved, preserve the query evidence, and route causal interpretation to the controller rather than inventing a narrative.

Case counts and multilingual questions

Public-sector scenario. A service manager asks how many cases aged past the applicable statutory limit by office. Reopened cases can cause double-counting if the query counts workflow events rather than reportable cases.

Use the case grain, elapsed-time rules, and threshold already approved for statutory reporting. The response should identify those rules, retain the original wording and SQL, and offer escalation to the reporting owner when exceptions affect the decision.

Global plant scenario. A supervisor asks an inventory question in Spanish that a colleague asked in English. Translation alone doesn't establish equivalent business meaning.

A multilingual text-to-SQL evaluation reported scores of 74.03 for English, 67.25 for Spanish, 58.54 for Vietnamese, 45.20 for Chinese, and 44.71 for Japanese using the same reported method. These are evaluation scores, not a guarantee for any enterprise deployment.

Employees can query SAP data in their own language when the implementation supports it and the glossary and acceptance tests cover that language. Map local abbreviations, currencies, fiscal calendars, and localized metric definitions to certified concepts; test indirect phrasing as well as literal translations.

  • Regional test sets should verify that equivalent questions produce the expected business result.
  • Audit records should retain the original language beside the generated SQL.
  • Human escalation should remain available for high-impact decisions in every supported language.

A credible pilot mixes these situations instead of relying only on straightforward historical lookups.

Governance Controls Before Anyone Acts

How does an answer stay reliable when definitions, security rules, and relevant context sit across SAP, warehouses, documents, and multiple SQL dialects? The operating rule is straightforward: the interface queries curated semantic views, not unrestricted operational tables.

The control checklist makes that boundary concrete.

A chart detailing governance controls for data reliability across SAP, warehouses, documents, and multiple SQL dialects.

Controls required for enterprise natural language querying

Control Failure it stops Minimum bar
Curated semantic views Queries against raw operational tables and wrong grain Only approved views and measures are reachable
Business glossary Two teams meaning different things by revenue or on-time Each metric maps to one certified formula
Row and column permissions A question widening what the BI tool allows Same roles as the existing analytics estate
Read-only default Generated statements that change data No data-changing SQL from the interface
Clarification rule A silent guess between competing definitions Ask when more than one certified meaning matches
Evidence with the answer A number nobody can audit Show tables, filters, aggregation, and freshness
Cost and timeout limits Runaway scans on large warehouses Warehouse budget and query-time guardrails
Language-specific tests A translation that changes the metric or calendar Regional test sets and retained original wording
Human review Forecasts or recommendations treated as facts Escalate high-impact decisions

The enterprise text-to-SQL benchmark paper documents the enterprise exact-match results in the scorecard and the difficulty of grounding queries in large schemas and distributed knowledge. The operational response is to manage business correctness explicitly, rather than treating successful SQL execution as approval.

Enforce controls outside the prompt

Use metadata retrieval to supply relevant object descriptions, approved relationships, and permitted values. Inspect sample values where necessary to distinguish similarly named columns, but apply the same access restrictions to samples as to full query results.

Parse and lint generated SQL before execution. Reject prohibited statements and references outside the approved object set; inspect query plans for unexpected joins and scans, then enforce warehouse-level budgets and timeouts.

Permissions must be enforced by the execution environment. An instruction telling the model not to disclose restricted information isn't a substitute for row- and column-level controls. Test equivalent questions under different identities, including requests that should return an access denial.

A refusal or clarification can be the correct answer. Don't train users to interpret every unanswered question as a system failure.

Monitor expected results, query-plan validity, authorization behavior, and clarification decisions together. Keep the question, its language, the resolved definition, generated SQL, and execution context in the audit record. Apply data governance for AI to ownership and change approval, so a revised metric or permission doesn't invalidate accepted behavior.

The accompanying video offers additional context for the governance discussion.

Before production approval, require representative questions with known results, explicit permission tests, and a written rule describing when the interface must ask rather than answer. Assign an owner to review failures and approve changes to the accepted question set.

Pilot Natural Language Querying on Governed Data

Deploy where the glossary, permissions, and semantic views already exist. Pause where the model would have to invent them. Decide that boundary before selecting a conversational interface.

A manageable first pilot covers a bounded business domain and certified metric family, with read-only access and production-like schemas. Historical retrieval belongs in that scope; unsupported predictions and operational changes don't.

Pilot scope verdict

Scope item Verdict Condition
Historical lookups on certified views In scope Glossary, permissions, and semantic views already exist
Unexplained forecasts Out of scope Until human review and the control list are met
Cross-system guesses Out of scope Until curated cross-system views exist
Any change to SAP or warehouse data Out of scope Read-only access only

Build the test set with a business owner, analytics engineer, and security representative. Include ordinary questions, alternative wording, ambiguous requests, restricted information, and cases where the correct behavior is to decline.

Score business correctness, not merely whether SQL ran. A passing response needs the expected value, approved definition, correct reporting period, appropriate authorization behavior, and an inspectable audit trail. Where a supported language is involved, assess the interpretation in that language rather than judging only an English translation.

Assess readiness before expanding

For SAP and Microsoft estates, inspect the semantic model, SAP-to-cloud data paths, refresh behavior, identity propagation, and existing reporting ownership. Resolve missing relationships and unclear metric definitions in the data platform, not through increasingly elaborate prompts.

We can assess those foundations against a proposed pilot scope. Our Generative AI solution works without replacing existing systems, allowing the conversation to focus on where governed access can be added rather than assuming wholesale platform replacement.

Expand only when reviewers can explain both successful answers and appropriate refusals. The acceptance question is simple: would the responsible business owner sign off on this answer and the method used to produce it?


Kagool combines SAP, Microsoft, and Databricks expertise with data-platform and governance services to help establish the foundations for trusted natural language querying. Visit Kagool to discuss an assessment of your semantic model, SAP-to-cloud data paths, and permissions, and identify a bounded, read-only pilot your business owners can validate.