AI Theatre vs Real AI

How to Tell the Difference

Part of A Sceptic's Guide to AI.

"AI-powered" has become the most overused phrase in technology marketing. It appears on products that use sophisticated deep learning and products that use a lookup table. It appears on genuine breakthroughs and on rebranded spreadsheets. The label has become so elastic as to be almost meaningless.

This is a problem if you are trying to make informed decisions about which AI products to adopt, which AI claims to believe, and where AI is genuinely transforming outcomes versus where it is decorating existing processes. Being able to distinguish real AI capability from AI theatre is one of the most practically valuable skills in the current technology landscape.

Contents

The Three Categories

When you encounter something labelled "AI," it typically falls into one of three categories.

1. Real AI capability

Systems where machine learning genuinely enables outcomes that would be impractical or impossible otherwise. The AI component is load-bearing — remove it and the product fundamentally degrades or ceases to function.

Characteristics:

  • The system handles complexity, variability, or scale that rule-based approaches cannot match
  • It improves with more data (and can demonstrate that it has)
  • It performs tasks that previously required human expert judgement
  • The creators can explain, at least at a high level, what the model does and what data it was trained on

Examples: Protein structure prediction (AlphaFold). Machine translation at scale. Medical image analysis that matches specialist radiologists. Commodity price forecasting using non-linear patterns across hundreds of correlated signals. Speech recognition that handles accents, noise, and context.

2. AI theatre

Systems that use AI branding, language, or interfaces without delivering AI-grade capability. The "AI" component is cosmetic — replace it with simple rules, search, or manual processes and the outcome would be the same or better.

Characteristics:

  • The product could achieve the same result with if-then rules or basic statistics
  • The "AI" is a chatbot wrapper around a FAQ or search index
  • No evidence of learning or improvement over time
  • The marketing emphasises "AI" prominently but the documentation is vague about what the model actually does
  • Removing the AI label would not change the product experience

Examples: "AI-powered" thermostat running a fixed schedule. Customer service chatbot that routes to a decision tree. "Smart" CRM that applies user-defined rules and calls them "AI insights." Enterprise analytics platform that computes basic statistics and labels them "AI-generated intelligence."

The one-line test. In AI theatre, the word "AI" is doing most of the work.

3. Good technology in the wrong context

Systems where the AI component genuinely works — the model is real, the predictions are sound — but it is applied to a problem where AI is not the bottleneck, or where simpler approaches would be more appropriate.

Characteristics:

  • The underlying technology is legitimate
  • But the problem it is solving does not require that level of sophistication
  • Or the output, while technically accurate, does not change any decision
  • Or the integration adds complexity without proportionate value

Examples: A sentiment analysis engine that tells you customers are unhappy — something you could learn by reading five support tickets. A natural language processing system that extracts dates from invoices when a regex would work at 99.9% accuracy for a fraction of the cost. A recommendation engine for a catalogue of twelve products.

Recognising AI Theatre

AI theatre is widespread because the incentives favour it. "AI-powered" commands higher valuations, higher prices, and more press coverage. The penalty for misusing the label is approximately zero. Here are the telltale signs.

The marketing-to-documentation ratio

Real AI products have technical documentation that explains, at some level, what the model does, what data it uses, and how it was validated. AI theatre products have extensive marketing pages and thin or non-existent technical documentation. If a product's AI claims live exclusively in sales decks and press releases, that is a signal.

The "any sufficiently advanced if-statement" test

Ask: could this product achieve the same result with hand-crafted rules? If the answer is yes — if the decision space is small, the inputs are structured, and the logic is deterministic — then the AI label is probably decorative. A scheduling tool that blocks your calendar during lunch is not AI, regardless of what the landing page says.

The improvement test

Real AI systems improve as they process more data. Ask the vendor: how has the system's performance changed over the last year? If they cannot answer — or if the answer is "we released a new version" rather than "the model improved through continued learning" — the AI component may not be doing what the marketing implies.

The removal test

Mentally remove the AI component. Does the product still work? Does it work roughly as well? If removing "the AI" leaves you with a functional product that has slightly less impressive marketing, you have identified theatre.

Recognising Real AI

Real AI capability tends to share several observable characteristics.

It handles variability. The system works on inputs it has not seen before — new text, new images, new market conditions — and produces reasonable outputs. It is not just matching against a known set of patterns.

It degrades gracefully. On inputs that are unusual or out of distribution, a real AI system produces outputs that are less confident or less precise, rather than silently wrong. (Not all real AI systems do this well, but the good ones do.)

It can be evaluated. The creators can point to benchmarks, validation datasets, or real-world performance metrics. They can tell you how the system performs on edge cases. They do not rely solely on cherry-picked demonstrations.

It solves a hard problem. The task requires handling complexity, ambiguity, or scale that makes rule-based approaches impractical. Translation across languages. Object recognition in varied lighting conditions. Predicting non-linear relationships in high-dimensional data.

Removing it breaks the product. Unlike theatre, where the AI is ornamental, in a real AI product the model is structural. Take away GPT from ChatGPT and you have nothing. Take away the ML from a commodity forecasting system and you have historical averages.

Good Technology, Wrong Context

This third category is the subtlest and in some ways the most costly, because it involves spending real money on real technology to solve the wrong problem.

The pattern usually looks like this: an organisation acquires AI capability (builds a team, buys a platform, or licenses a model) and then looks for problems to apply it to. The technology works. The model performs. But the business problem it is pointed at either does not require that level of sophistication, or the AI output does not connect to any decision that changes outcomes.

Common variants:

  • Insight without action. The AI produces accurate analysis that nobody acts on because the insight was already known, the organisation lacks the process to respond, or the analysis is too abstract to be actionable.
  • Overkill. The problem has a simpler solution that works at 95% of the accuracy for 5% of the cost. Using a large language model to parse structured data that conforms to a known schema. Training a neural network to classify documents into three categories.
  • Missing integration. The AI component works in isolation but is not connected to the systems where decisions are actually made. A brilliant demand forecast that sits in a dashboard nobody checks because purchasing decisions are made in a different system using last month's spreadsheet.

The tell: ask what decisions change because of the AI component. If the answer is vague or aspirational — "it gives us better visibility" or "it enables data-driven decision-making" — the technology may be real but the deployment is theatre.

The Evaluation Checklist

Use these questions when evaluating any AI product or initiative. They work whether you are assessing a vendor pitch, an internal project proposal, or your own team's work.

  • What specific task does the AI perform? If the answer is vague ("it uses AI to optimise…"), probe until you get a concrete answer or confirm there is not one.
  • What data does the model use, and where does it come from? Real AI requires real data. If the data story is unclear, the AI claim may be inflated.
  • Could this be done with rules or basic statistics? If yes, the AI component may be adding cost and complexity without proportionate value.
  • How is performance measured? Real AI products have quantitative benchmarks. "It works really well" is not a metric.
  • Has performance improved over time? One of the defining characteristics of AI is that it learns. If the system has not improved since deployment, it may be static rules wearing an AI label.
  • What happens when the AI is wrong? Products built around real AI have error handling, confidence scores, or fallback mechanisms. Products that never discuss failure modes are selling certainty they do not have.
  • What decision does this change? AI that does not connect to a decision is expensive decoration. If the output does not alter what someone does, it is not delivering value regardless of how sophisticated the model is.
  • What happens if you remove the AI component? If the answer is "not much," you have found theatre. If the answer is "the product stops working," you have found real capability.

Key takeaways:

  • Not everything labelled "AI" delivers genuine AI capability — learn to distinguish real AI, AI theatre, and misapplied AI
  • Real AI handles variability, improves over time, can be evaluated, and is structural to the product
  • AI theatre uses the label for marketing without the underlying capability — test by mentally removing the AI component
  • Good technology in the wrong context is costly because the AI works but does not change decisions
  • The single most important question: what decision does this change?

Related Reading

I help organisations evaluate AI products and separate substance from spin. If you are making AI adoption decisions, get in touch.

Written by Dr Tristan Fletcher.