Notes from a Data Leader:

Why AI Costs More Than It Should

Token Prices Aren’t the Whole Story

It seems obvious: as token prices fall, AI should get cheaper. But it hasn’t worked that way.  

Most leaders I speak with are watching their AI bills climb even when the cost per token is decreasing. It feels backwards, but it isn’t. There’s a hidden factor that is driving AI costs.

It’s the number of tokens it takes to complete a task.

Regardless of the cost per token, the number of tokens required to complete a task has exploded. The reason has nothing to do with the model. It has to do with the data underneath it.  

Every Step Costs Tokens

AI agents don’t just answer questions anymore. They plan, call tools, split tasks with other agents, verify their own outputs, and retry when something goes wrong. Every one of those steps cost tokens, and a lot of the cost goes to figuring out things that humans already know: the meaning of a specific column, which numbers can be trusted, who’s allowed to see it.

Finding this information for an agent isn’t a one-time cost. It happens repeatedly with every call, because nothing about the meaning of the data travels with the data itself. The agent starts from zero every time.

What We Can Control

To optimize costs, people often give the most attention to model pricing and orchestration design. But it’s data leaders who control how the data is organized before an agent even touches it.

Think about a warehouse where every box arrives unlabeled. A worker can’t do anything with the box until they open it, analyze what’s inside, find who it’s for, note if it’s fragile, and figure out where it’s supposed to go. Now imagine doing this for every box, every time someone new comes across it, and the warehouse becomes highly inefficient. This isn’t because the workers are slow; it’s because zero information about the box is communicated.

Enterprise data mostly ships unlabeled. A table in a data lakehouse doesn’t specify what “net revenue” means, which fields are trustworthy, or who’s allowed to see them. An AI agent querying that table must rebuild all of that understanding with every call. ‍

What Changes When Data Carries Its Own Context

The fix isn’t a smarter model. It’s giving the agent data with meaning already attached. That’s what a governed, contextualized data product does: the definitions, quality signals, lineage, and access rules travel with the data instead of in an old wiki that nobody updates.

When the context is built once and reused, you stop paying for your agent to rediscover it with every call. Instead of repeatedly sending definitions, business rules, and documentation in every prompt, the agent can retrieve structured context as needed. Teams using this approach have reduced prompt token consumption by as much as 90% compared to agents working from undocumented schemas. Over time, those unnecessary tokens become the difference between an AI program that scales economically and one that quietly becomes too expensive.

It’s Not Just the Data. It’s Memory Too

There’s a version of this problem that goes a layer deeper than static tables. What happens once agents start accumulating their own memory over time? Call it memory debt: the growing gap between how much an agent’s memory holds and how well anyone can inspect, correct, and explain it.

Jorge Luis Borges' character Funes the Memorious is the perfect example. He is cursed with perfect recall, able to remember every leaf on every tree. But he couldn’t think, because thinking depends on forgetting and deciding what no longer matters. An agent with unmanaged memory has the same problem. It has infinite retention, no judgment, and no architecture for letting go. That's just a new, faster-growing version of the old data swamp, except now it's built inside the agent instead of inside a lake.

The fix is the same: don’t let context accumulate by default, govern it on purpose. What matters, what’s stale, what should be ignored or forgotten, that has to be a design of the system. Structure is what makes it possible to add, remove, and replace parts of a system without losing what the system is for. Data foundations give agents that structure. Without it, more memory just makes an agent more expensive and harder to trust.

The Takeaway

Cheaper tokens are not the solution. And the next model discount isn’t either. The highest-leverage decision data leaders can make right now is ensuring the data behind each task is organized well enough that context travels with it. That way, the agent doesn’t have to start from scratch with each call.

Getting the foundation right will allow today’s agents and everything built on top of it to be cheaper, faster, and more reliable.

Until next time,  
Saurabh