Gartner has predicted that explainable AI (XAI) will play an increasingly central role in the deployment of generative AI, with investment in LLM observability expected to rise significantly over the next several years.

LLM observability refers to the tools and practices used to monitor, analyse and evaluate the behaviour of large language models in production, including issues such as hallucinations, bias and output quality.

According to the firm, by 2028, spending on LLM observability will account for 50% of GenAI deployments, up from 15% today.

Gartner describes XAI as a set of capabilities that enables users to understand how AI models operate, including their strengths and weaknesses, predicted behaviours, and potential biases. These capabilities are designed to support accuracy, fairness, accountability, stability, and transparency in algorithmic decision-making by clarifying how outputs are generated for specific audiences.

Alongside explainability, LLM observability tools are becoming increasingly important in monitoring and evaluating the performance of AI systems.

These solutions provide insights into how models behave in real-world environments, going beyond traditional IT metrics such as response times to assess factors including hallucinations, bias, and token usage. They are used not only by development teams but also by IT operations and site reliability engineers responsible for maintaining system performance in production.

“As enterprises scale GenAI, the trust requirement grows faster than the technology itself,” said Pankaj Prasad, Sr Principal Analyst at Gartner.

“XAI provides visibility into why a model responded a certain way, while LLM observability validates how that response was generated and whether it can be relied on.

“Without robust XAI and observability foundations, GenAI initiatives will be restricted to low risk, internal, or noncritical tasks where output verification is easily managed or inconsequential, severely limiting the potential return on investment.”

The analyst firm also forecasts rapid growth in the generative AI market, estimating that the global GenAI models market will exceed $25 billion in 2026 and reach $75 billion by 2029. This expansion is expected to drive further demand for tools and frameworks that can verify AI-generated outputs and mitigate risks such as hallucinations, factual inaccuracies, and biased reasoning.

As organisations move from experimentation to production-scale deployments, the metrics used to evaluate AI systems are also evolving.

Prasad added: “Traditional observability is focused on speed and cost, but the priority is now moving toward deeper quality measures such as factual accuracy, logical correctness and sycophancy. This shift requires new governance-focused metrics and evaluation methods, such as human-in-the-loop validation of the generated content’s narrative and citation accuracy.

Recommended reading

• Scot Gov Reveals Scotland’s AI Strategy
• Scotland’s Techscaler Startups Smash £257m Funding Milestone
• Scotland Publishes 5-year AI Strategy to Compete Globally
• Scotland Leads UK in AI Productivity Gains

He added that explainability and observability are foundational to the long-term viability of generative AI systems. “Explainability turns a GenAI output into a defensible, auditable insight. LLM observability ensures the model behaves as expected over time. Without both, GenAI cannot mature beyond controlled lab environments.”

Gartner advises organisations to take a structured approach to strengthening trust in their generative AI deployments.

This includes implementing explainability tracing for high-impact use cases to document model reasoning and data sources, adopting observability platforms capable of tracking performance and quality metrics, integrating continuous evaluation into CI/CD pipelines, and ensuring that legal, compliance, and other stakeholders are aligned on explainability requirements and governance expectations.

Together, these measures are expected to form the foundation of more transparent, accountable, and scalable generative AI systems as adoption continues to accelerate across industries.