AI in practice – The number looked retrieved. It was Generated

An AI assistant in a financial app I was building told me I already owned 250 shares at $5.08. The position didn’t exist. If I hadn’t known my own portfolio, I would have believed it.

It sounded retrieved. It was generated.

While working on an AI-assisted financial application, I ran into a failure that changed how I think about grounding.

The model flagged what it believed was an existing position: “You already have 250 shares at $5.08.”

It was specific enough to sound like a database lookup. But I knew the position didn’t match what I expected, so I asked where the number came from.

The model checked the database. There was no such record. Then came the response: “I made up the 250 shares at $5.08 number.”

We know LLMs hallucinate. That wasn’t the interesting part. What bothered me was the precision. 250 shares. $5.08. Those don’t look like guesses. They look retrieved.

And even the confession deserves some skepticism. “I made up that number” is generated text too. It’s a plausible explanation of what happened, not an audit trail of why it happened.

My first fix was to strengthen the instruction:

Never state portfolio data without checking the database first.

That’s useful. But it’s not a control. The model already had access to the data. The problem was that nothing required it to retrieve that data before producing an answer. Adding another instruction still leaves the same probabilistic system responsible for deciding when to verify itself.

So I started changing the architecture instead. Financial facts come from the underlying data sources. Calculations happen in deterministic code. Investment rules are applied against those calculated facts. The LLM moves downstream.

It can help explain what the system found and why it matters. But it doesn’t get to establish the underlying numbers. Figures shown to the user come from the deterministic data layer, not from the model’s prose.

If the model disappears entirely, the application should still know what you own, what it’s worth and why a rule was triggered. The AI becomes an interpretation layer, not the source of truth.

That requires more plumbing, stricter interfaces and sometimes a system that says, “I don’t have that information,” instead of smoothly filling the gap. I’m increasingly comfortable with that tradeoff. Precision is not provenance.

The goal isn’t to build systems where LLMs never hallucinate. It’s to build systems where, when they do, they cannot redefine the truth.

One question worth asking this week

What should your AI systems be allowed to reason about, but never be allowed to establish as fact?

If your system can’t say where a number came from, neither can your users.

If your team has run into something similar, or found a different way to handle it, I’d genuinely love to hear about it. Just reply.