How NASA Built a Tool-Grounded LLM for Safer Air Traffic Decisions

The real problem was never conversation

Air traffic operations are messy in the least cinematic way possible. Not dramatic. Just dense.

Decision-makers deal with changing weather, runway constraints, advisories, route changes, live flight feeds, and the usual human issue: nobody stays perfectly sharp forever. Add advanced air mobility into a more crowded sky, and the cost of ambiguity gets expensive fast.

A plain LLM is attractive here because people can ask natural-language questions. But that same convenience comes with a flaw aviation can’t casually tolerate: hallucination.

NASA’s design choice: route first, answer second

Based on the study description, the NASA Ames team tackled hallucination as a system design problem, not a “make the model smarter” problem.

Their setup appears to work like this:

• A user asks a natural-language aviation question
• The LLM interprets intent and extracts key entities
• It selects one or more validated aviation tools
• It fills in inputs using defaults or helper plugins when needed
• It runs those tools against current or structured data
• It returns a natural-language response that explains what it used

That is a very different job from open-ended generation. The model is less oracle, more dispatcher.

And in a safety-sensitive setting, dispatcher is the healthier personality. This is closely related to tool calling.

What “tool-grounded” means in practice

The system reportedly connects to nine aviation-validated tools and data sources. These include things like:

• trajectory prediction models
• runway configuration logic
• FAA document retrieval
• live weather sources
• route APIs
• traffic management advisory sources
• NASA live operational data services
• an air traffic control simulator

The key idea is simple: when the answer depends on a current condition or a domain-specific rule, don’t let the model improvise.

If someone asks for the best runway configuration at JFK “right now,” the answer should not come from a model’s memory of general aviation patterns. It should come from current weather inputs plus a runway decision model built for that purpose.

That’s not just grounding. It’s accountability with better UX, especially in air traffic control.

Why this reduces hallucination risk

A standard LLM often fails in two familiar ways:

• it answers confidently when it should decline
• it blends partial knowledge into a plausible fiction

The NASA approach seems designed to interrupt both.

Because the model is routing to verified tools, the response is tied to actual data sources and actual operational models. Because the system can also say “no information found” or decline out-of-scope requests, it avoids the classic LLM urge to be helpful past the point of competence.

That second part is underrated. In critical workflows, refusal is often a feature, not a bug.

The benchmark result worth paying attention to

The reported performance numbers are strong, but the more useful insight is where performance changed.

According to the provided context, the stronger model achieved high tool-selection and input-selection accuracy across the full toolset, and also performed well on a separate held-out test set. That suggests the routing layer was not only tuned for a toy demo.

But the more practical takeaway is this: smaller models seemed to hold up reasonably well for looser, open-ended tool calls like weather, routes, live data, and manual lookups. The gap widened when the task required strict scope control.

That makes sense. It’s one thing to call a weather source. It’s another to recognize, “This runway model only supports a few airports, so I should decline.”

In other words:

• open retrieval is easier
• disciplined refusal is harder
• safety often lives in that gap

Where the system still struggled

One recurring weakness was weather-related queries.

That’s a useful reminder that “tool access” does not magically solve judgment. A model still has to decide whether a question actually requires the weather tool, or whether it can answer directly, or whether it should decline. General meteorological questions can blur those boundaries.

This is where many enterprise AI projects get awkward. People focus on adding tools, but the hard part is often governing when not to use them.

Multi-tool workflows are where this gets interesting

Single-tool calling is nice. Multi-tool coordination is the real test.

The examples in the study description suggest the system could carry forward context between steps. If a user first asks about weather and then asks about runway configuration, the system can reuse the weather data instead of starting over. In another scenario, it can fetch an FAA advisory, then use that context to inform route recommendations.

That matters because real work is rarely one clean query. It is usually:

• get the latest condition
• apply the rule
• check the route
• explain the recommendation
• show assumptions

This is exactly where Multi-tool coordination has an edge over “just ask the chatbot again.”

The baseline comparison says a lot

The plain LLM baseline reportedly did fine on some static regulatory questions. No surprise there. General models are often decent at text recall, until they’re very much not.

But on real-time or structured operational tasks, a model without tools is boxed in. It cannot see current advisories. It cannot fetch live weather. It may also struggle with structured filtering over flight data.

That split is a helpful buying signal for anyone evaluating AI systems:

• if your workflow depends on current data, tool grounding is not optional
• if your workflow depends on exact domain constraints, refusal behavior matters
• if your workflow is mostly static reference lookup, a plain LLM may look better than it really is until one bad answer matters

The hidden cost: human setup

This system did not emerge from one clever prompt and a victory lap.

The team reportedly needed many revision cycles with expert feedback before prompt behavior stabilized, especially for early tools. That is a sober detail, and a useful one.

Tool-grounded AI is often sold as clean architecture. In practice, it is also policy work. You need experts to define boundaries, review failure modes, and decide what “correct” means when the question is ambiguous.

The model may write the sentence. Humans still write the rules.

Why this matters beyond aviation

The broader lesson is bigger than FAA data or runway models.

In high-stakes domains, the strongest LLM pattern may not be “ask the model more cleverly.” It may be:

• narrow the role
• connect it to validated tools
• expose assumptions
• make refusal normal
• separate fluent language from factual authority

That applies to healthcare, finance, legal operations, industrial systems, and pretty much any workflow where an elegant mistake is still a mistake.

The model is useful not because it knows everything, but because it knows which system should answer.

What AI tool buyers should take from this

If you’re comparing AI tools for serious operational use, don’t stop at demo quality. Ask uglier questions.

Look for answers to things like:

• Does the system call verified tools or mostly generate from memory?
• Can it use live data sources?
• Does it explain what source or model produced the answer?
• Does it decline out-of-scope requests reliably?
• Where does a smaller model work, and where do you need a stronger one?
• How much expert tuning was required to make it trustworthy?

That checklist is less glamorous than “AI copilot.” It is also more useful.

The useful takeaway

NASA Ames’ approach suggests a practical rule for safety-critical AI: the LLM should not be the expert. It should be the switchboard.

If a tool can pull verified data, run a validated model, and show its assumptions, let the tool do the hard part. Let the LLM handle the interface, the routing, and the explanation. In domains where a wrong answer can ripple outward, that kind of humility is not a limitation. It’s the product.