The Hard Questions Enterprise Leaders are Asking About AI

<span class="text-italic body-24">What we heard from 11 leaders at our recent roundtable on context, cost, enterprise data, and putting AI to work.</span>

Enterprise AI has moved well beyond the “what can we do with it?” stage. The harder questions now are about what it takes to make AI useful inside a real enterprise: the data it relies on, the context it needs, the cost of running it, and the decisions it ultimately needs to support.

Those questions look very different when you put leaders from different parts of the business around the same table.

That’s what we did at our recent Top 1% Leaders Roundtable in Mumbai, bringing together 11 senior leaders across data, technology, digital, security, and supply chain. Dinker Charak, Head of Solutions Engineering at The Modern Data Company, moderated the discussion, opening with a simple question: How would you describe the current state of enterprise AI and data in one word?

The responses ranged from ubiquitous and chaotic to precious, secure, and valuable. From there, the conversation quickly moved to the harder questions: What happens when AI meets the reality of enterprise data? What does it need that today’s systems don’t provide? And where should organizations focus if they want to move beyond experimentation?  

AI is exposing how much humans have always filled in

For years, enterprise systems have relied on people to supply context that isn’t captured in the data itself. An analyst knows that two customer records refer to the same relationship. A supply chain leader understands why a delay in one system matters to an order in another. A local team knows that the same term can mean something different depending on the market.

AI doesn’t come with that institutional knowledge. That distinction came up a lot during the roundtable. Giving AI access to more enterprise data doesn’t necessarily give it more understanding. An AI system may be able to retrieve a customer record, a contract, a transaction, and a support history without knowing how those pieces fit together, which source to trust, or what matters for the question being asked.

Localization was one example raised in the room. In a market as varied as India, language is only one part of the equation. Region, terminology, customer behavior, and business practices can all affect what data means. A general-purpose model doesn’t automatically understand the conventions of a particular company, market, or process.

Unstructured data raises a similar challenge. Enterprises have enormous amounts of knowledge sitting in contracts, documents, emails, transcripts, and other content. Making that information available to AI is one thing. Connecting it to the right customer, transaction, policy, product, or business process is much harder.

That’s where context becomes important. A context layer carries the definitions, relationships, metadata, quality, governance, history, and operational state AI needs to interpret enterprise data. Instead of expecting every model or application to reconstruct that understanding, the context can be built once and reused.

How much work are we pushing onto the AI?

Prompt engineering also came up during the discussion. Good prompts matter. So do guardrails. But there’s a limit to how much business knowledge we should expect a prompt to provide.

Every time someone has to explain an internal term, specify which source to trust, describe how two datasets relate, or provide the business rules behind a request, they’re giving the AI context the organization already knows. Then the next user, agent, or application may have to provide it all over again.

There’s a cost attached to that. When data isn’t prepared for AI, one response is simply to give the model more: more data, more documents, more retrieval, larger context windows, more tokens. The model ends up spending time finding, interpreting, reconciling, and filtering information before it can get to the work the user asked it to do.

The roundtable discussion raised an important question as organizations start looking more closely at AI costs: How much of the work we’re paying AI to do could be handled in the data layer instead?

That changes the economics. AI efficiency isn’t only about model or token prices. It’s also about reducing the amount of unnecessary work required to get from enterprise data to a useful answer or action. It can also open the door to using smaller, more targeted models when a large general-purpose model isn’t necessary.

Start with the decision

One of the simplest questions raised during the roundtable may also have been one of the most useful: What decision are we trying to make, and what data does AI need to make it?

That sounds obvious. In practice, the rush to deploy AI can lead teams in the opposite direction: connect the model to as much information as possible and let it figure out what matters.

Starting with the decision creates a much clearer path. What information does the AI need? What relationships need to be understood? Which definitions have to be consistent? What data can it use? What level of quality is required?

The answer might lead to a highly sophisticated agent. Or it might lead to something much simpler: a better recommendation, a more accurate demand forecast, a faster supply chain decision, or an existing BI application that can answer questions that previously required an analyst.

The goal is the outcome, not the amount of AI involved.

The existing data stack still matters

The discussion also made clear that preparing data for AI doesn’t mean starting over. Enterprises have spent years investing in warehouses, lakehouses, operational applications, governance tools, and pipelines. Those systems continue to do important jobs.

What changes with AI is what organizations need from the data across those systems. AI needs to know what the data means, how it relates, which data can be trusted, how it can be used, and what rules apply. And that understanding needs to be available consistently across models, agents, applications, and analytics. That’s the role of the context layer.

The conversation in Mumbai started with a question about how leaders would describe enterprise AI today. It ended up somewhere more useful: with a discussion about what it takes to make AI work inside the complexity of a real enterprise.

The models will keep changing. The use cases will keep changing too. The work now is making sure enterprise data is ready for what comes next.