Measuring AI Search: What Actually Matters — AEO Guides

Most marketing teams measure AI search like a performance channel. They track clicks, conversions, and "visibility scores" as if it were paid search. This is a category error. AI search operates in the mental-availability space, not the direct-response bucket — and when you measure it wrong, you optimise for the wrong thing.

The honest assessment: AI referral traffic is roughly 1% of total web traffic, and Google AI Overviews show a 93% no-click rate. Yet the traffic that does arrive from AI converts at roughly 15.9% — about 9x the rate of Google organic. The volume is low; the quality is high. The real impact is brand influence that happens long before a click.

This guide separates signal from noise. We'll cover why most vendor metrics fail, which metrics are actually useful, and a practical five-level framework your team can use today.

Why Common Metrics Mislead

LLMs are non-deterministic

The same prompt rarely produces identical output. Temperature randomness, mixture-of-experts routing, and floating-point arithmetic mean that running a prompt 100 times can return your brand in a different position almost every time. "You rank #3 for X prompt" is noise, not signal.

Most tools do not sample enough

To measure brand mention frequency with statistical validity, you need 60–100 runs per prompt. Many tools run 100–200 prompts once each and call it a metric. It is a snapshot of randomness.

Citation count without context is meaningless

Appearing in 150 AI responses tells you nothing about whether those mentions were positive, high-intent, or influential. A single authoritative mention in a high-intent response beats 50 low-quality ones.

"AI visibility scores" are non-standard

Each vendor defines visibility differently. Some weight by query volume, some count links, some count direct mentions. A score of 73 from one tool is not comparable to 73 from another.

Single-touch attribution is impossible

Purchase journeys take an average of 79 days and involve 56 brand touchpoints. The person who clicks from a Google result in May may have first seen your brand in an AI response in February. Last-click attribution arbitrarily assigns credit to the final touchpoint and ignores the rest of the chain.

The Five-Level Measurement Framework

Start with what is measurable and actionable. Add more sophisticated layers only when the foundations are solid.

Level 1: Technical Accessibility

Question: Can AI agents access your content at all?

This is binary and deterministic. If AI cannot reach your pages, nothing else matters.

Metrics:

Percentage of key pages accessible to AI agents
robots.txt blocks on AI crawlers
Critical content rendered in HTML

How to measure:

Run a Compass audit
Manual test: ask ChatGPT/Claude to find key pages
Use Screaming Frog to compare raw HTML vs rendered DOM

Level 2: Content Quality and Accuracy

Question: Does your content actually answer the questions people ask AI?

Metrics:

Percentage of key facts correctly extracted by AI
Significant inaccuracies found during review
Freshness of evergreen content

How to measure:

Ask AI to summarise your pages and check the output
Test extraction of specific facts
Maintain an audit trail of content updates

Quality beats quantity. One accurate, comprehensive answer is worth ten shallow mentions.

Level 3: Brand Mention Frequency

Question: How often does AI mention your brand?

This is measurable only with discipline. Position is noise; frequency over a large sample is signal.

Metrics:

"Brand appears in X% of relevant prompt responses (±5%, n=60 runs per prompt)"

How to measure:

Use a tool like SparkToro or Gumshoe, or build your own prompt tracker
Run 60–100 identical prompts per brand/category
Track trends over time, not absolute values

Level 4: Consideration Set / Brand Recall

Question: Do people remember your brand better after seeing it in AI?

This measures mental availability — the real value AI search creates.

Metrics:

Aided brand awareness
Consideration-set inclusion
Preference shift vs competitors

How to measure:

Brand tracking studies (YouGov, Tracksuit, Latana)
Pre/post surveys in your category

This is how TV was measured for decades. It accounts for time lag and multi-touch influence.

Level 5: Purchase Attribution

Question: Does AI visibility drive sales?

Direct attribution is structurally impossible. The best available approach is Marketing Mix Modelling, which estimates contribution over time using statistical techniques.

Metrics:

Estimated annual revenue contribution with confidence intervals

How to measure:

MMM study with 12+ months of data
Cost: £20–50k+ and statistical expertise

Be honest with stakeholders: this is an estimate, not a measurement.

The Traffic Reality

To set expectations, understand the scale:

Google drives ~40% of website traffic.
ChatGPT drives ~0.21% of website traffic.
Overall AI referral traffic is ~1% of website visits.
Google AI Overviews have a ~93% no-click rate.

At current scale, AI search will not materially move revenue for most businesses. For high-consideration B2B, professional services, or premium e-commerce, the small volume of qualified traffic can still matter. The trajectory is uncertain, but the conversion quality is real.

The ROPO effect applies: users research on AI, then buy on Google or offline. Only the final touchpoint is tracked, which makes single-touch attribution misleading.

What to Stop Reporting

Remove these from your dashboard:

Prompt position / AI rankings — position is random variation
Share of voice / AI visibility index — non-standard, incomparable
Citation count without context — count alone is meaningless
Single-touch attribution for AI — structurally invalid
"AI traffic" as the main metric — too small and ignores brand influence

What to Report Instead

Replace them with:

"95% of key pages are accessible to AI agents" (Level 1)
"89% of key facts correctly extracted by AI" (Level 2)
"Brand appears in 45% of relevant AI responses (±5%, n=60)" (Level 3)
"Aided brand awareness up 3pp since baseline" (Level 4)
"We estimate AI contributes £50–70k annually (±20%, 95% CI)" (Level 5)

Recommended Measurement Roadmap

Week 1–2: Run a technical accessibility audit (Level 1).

Weeks 2–3: Complete a manual content-quality audit (Level 2).

Month 2–3: Set up brand-mention tracking if budget allows (Level 3), or build a basic prompt tracker yourself.

Quarter 1: Establish a brand-tracking baseline (Level 4).

Ongoing: Re-run Level 1 and 2 quarterly to catch regression.

Annually: Conduct a brand lift study or MMM analysis if you have the scale (Level 5).

Summary

AI search is not a performance channel. It is a brand-building, consideration-shaping medium that influences buyers long before they click. The right measurement starts with foundations — can AI access and accurately extract your content? — then moves up to brand recall and estimated business impact. Most teams are measuring the wrong thing because the wrong thing is easier to sell. Measure what is real instead.

Start with Level 1: Compass tests whether AI agents can access and navigate your site in minutes.