How AI Agents Navigate Websites: What 3,348 Tasks Reveal — AEO Guides

AI agents don't browse like humans, and they certainly don't crawl like search engines. They navigate in real-time, making semantic judgments about which links to click to complete specific tasks. Understanding how they navigate is fundamentally different from understanding SEO ranking.

Our research examines how AI agents navigate websites, where they succeed, and where they fail. Using 3,348 navigation tasks across 269 websites, we identify patterns that challenge conventional assumptions about AI-optimised content. The headline success rate is 78.6%, but this aggregate hides critical variation across industries and entry points. The goal is not to present this as "good" or "bad," but as a baseline to explain why specific patterns emerge.

For SEO professionals, the takeaway is distinct: AI navigability is a separate problem from SEO. Traditional signals like domain authority or backlink profiles do not guarantee an agent can complete a task. You need to understand the mechanics of navigation—entry strategies, link selection, and information architecture—separately from crawling.

What We Tested

To understand AI navigation, we needed a dataset significant enough to cover variation but structured enough to isolate variables. We measured 3,348 navigation tasks across 269 websites, generating 494,197 link evaluations. This sample size is large enough to be statistically significant and broad enough to cover industry variation, including 77 task types across 7 sectors.

A "task" is defined as a specific objective an AI agent attempts to complete. This includes finding pricing information, locating contact details, or identifying a returns policy. The AI agent starts at a given entry point and navigates the site to find the target content. Success is binary: the agent found the target or it did not. Failure is defined as the agent giving up or exhausting its maximum step limit.

The dataset includes a specific breakdown by industry to ensure representation. SaaS accounted for 648 traces, E-commerce for 497, with the remainder distributed across Financial Services, Content/Media, Enterprise/B2B, and Pharma/Healthcare. This ensures findings apply across different site structures, not just a single vertical.

We tested three distinct entry strategies to understand how the starting point influences success:

Homepage-first: The AI starts at the site homepage and navigates from there.
Search-first: The AI searches Google, clicks the first result (regardless of domain), and navigates from that landing page.
Hybrid: The AI searches Google but filters results to the target domain only.

Entry strategy matters because it determines the starting position. A landing page might contain the answer (instant success) or require navigation (multi-click task). We measured link scoring and click selection, tracking clicks post-landing while tracking instant successes separately. Of the 3,348 tasks, 37% were instant successes (0 clicks), while 1,999 traces required active navigation. Success rates varied by entry strategy: Hybrid achieved 95%, Homepage 76%, and Search-first 64%. Understanding this methodology is critical to interpreting the findings that follow.

Entry Strategies: The Search Paradox

Search is often assumed to be the most reliable path for AI discovery. However, our data reveals a search paradox: search-first entry is bimodal, resulting in either instant success or total collapse. While the headline success rate is 64%, this masks a crucial split between instant resolutions and multi-click failures.

When search works, it provides the "search shortcut." In 59% of search-first attempts, the AI lands directly on the answer with zero clicks needed. In these cases, the success rate is 90%. This happens when a query like "Stripe pricing" returns the actual pricing page as the top result. The AI lands, identifies the content, and completes the task. This efficiency is why many AI browsing tools prioritise search-first strategies; when the landing page is correct, the task is done immediately.

However, when search fails, success rates collapse to 27%. This occurs when the AI lands on a suboptimal page. If the Google search result is a blog post about pricing rather than the pricing page itself, the AI must navigate. This is where the problem begins. Google optimises for user engagement and click-through rate, not AI task completion.

Specific examples from the research illustrate this intent mismatch. A search for "ahrefs.com product roadmap" often lands on an "SEO Roadmap" blog post, not the actual product page. A search for "Stripe pricing" might land on a comparison site. Of the failed search-first traces, 17% forced the agent back to the homepage, 7% landed on blog posts about the topic, 5% on legal or privacy pages, and 4% on 404 errors.

This collapse happens due to keyword collision and landing page intent mismatch. For instance, "product roadmap" and "SEO roadmap" have semantic overlap, causing content about SEO roadmaps to rank for commercial intent queries. Modern SEO often optimises for SERP snippets (zero-click answers), not page content, leaving the AI with a landing page that summarises the answer without the full context required for navigation.

The hybrid strategy eliminates this problem. By filtering search to the target domain only, success rates rise to 87% (vs. 64% for naive search-first). When starting on the correct site, success shoots up. This suggests that while Google's ranking model might not optimise for AI, ensuring the agent lands on the correct domain is the primary lever for success.

For practitioners, this implies that ranking #1 for a keyword does not mean the site is AI-optimised. Two AI agents searching for the same thing might land on completely different pages—one on the product page, one on a blog post. Traditional SEO success does not equate to AI discoverability. If you rely solely on search-first entry, you are subject to Google's volatility.

The Two-Click Rule: Why Depth Kills Navigation

Success drops sharply after two clicks. This is the most consistent pattern in our data and has profound implications for site architecture. We measured success rates by the number of clicks required after landing on the starting page.

0 clicks (instant): 99% success
1 click: 89% success
2 clicks: 46% success
3+ clicks: 28% success

Visualise this as a sharply declining curve. The drop from 1 click (89%) to 2 clicks (46%) represents a critical threshold. Beyond two clicks, failure becomes more likely than success.

This pattern emerges because each click has a probability of error. When an agent successfully completes a task, it makes the correct click roughly 70% of the time. Compounding this error over three clicks (0.70^3) predicts a cumulative success rate of roughly 34%, which aligns almost perfectly with our observed data for 3+ clicks.

It is important to define what "click" means in this context. The count starts after landing on the starting page. Zero clicks means the landing page was the answer. One click means the agent landed, clicked one link, and found the answer. This matters because 37% of successful tasks complete with zero clicks. When measuring depth, we focus on the multi-click traces (1,999 of 3,348 tasks).

The implication is clear: critical pages should be close. Homepage navigation is relatively consistent at 76%, but critical content buried three clicks deep will fail 70%+ of the time. If AI needs to find something, two clicks is the maximum depth threshold for reliable success.

Common failure patterns at depth reveal the mechanism. 95% of failures involve loops, meaning the agent revisits a page it has already seen. This suggests unclear navigation options, not exhausted choices. The median page has 139 links, so the agent has ample options; it is the clarity of those options that dictates success. Poor link semantics create the appearance of new options when links actually lead to the same page.

Architecturally, this means flattening important content. Deep category structures become invisible to agents. Nested product taxonomies fail unless clearly labelled. If your pricing page is three clicks from the homepage, it is effectively invisible to AI agents relying on navigation.

Position Trumps Relevance

A surprising finding from our research is that position matters more than relevance. AI agents exhibit a strong positional bias, clicking links in the top 50 DOM positions 80% of the time. The median clicked link position is 45, while the median non-clicked link position is 116.

However, success rates are essentially flat (68-74%) whether the link is at position 10 or position 200. This means the agent has learned positional bias from training—scanning top-to-bottom like humans—but lacks the contextual reasoning to know when a lower link is actually the right choice. Position is a learned heuristic, not an optimisation.

Where do agents click? 80% of clicks target navigation elements (header, footer, sidebar). Body-text links are largely ignored. Agents correctly infer that navigation is the "escape route" to find other pages, whereas body text is about the current page. This has practical implications for internal linking.

Critical links should be in the header or navigation. Footer links are effectively invisible to AI agents. Navigation is not just useful for humans—it is essential for AI. In-body links are dead to AI agents regarding navigation; they might exist for context, but they do not drive the agent forward.

Semantic similarity of link text matters, but only if the link is in a position the AI will evaluate. Clarity does not compensate for placement. This creates a paradox with classic SEO advice. Traditional SEO suggests "putting important links everywhere" to drive crawl budget and authority. The AI reality is to put important links in nav where agents will click them. These aren't entirely in conflict, but the emphasis differs.

Link density also impacts findability. Pages with multiple links to the same destination confuse agents. The median page has 65% of links sharing a URL with at least one other link. An agent might think "Products" and "Our Solutions" are different options, but both lead to /products. This creates the illusion of navigation options when there aren't actually any.

To optimise for position, place critical content in the navigation menu. Ensure link text is semantically distinct. Avoid duplicating links to the same destination unless necessary for clarity.

When Navigation Fails: The Loop Trap

When AI navigation fails, it is rarely because content is inaccessible. It is because the navigation structure is confusing and looping. Our data shows that 95% of failures involve loops, where the agent revisits a page it has already seen. This is not the same as "no way forward." The median page has 139 links, so the agent literally has options; it is stuck circling.

The typical failure sequence is consistent. The AI clicks a link, finds a new page without the target content, evaluates links to navigate elsewhere, and one of those links leads back to a previously visited URL. The loop is detected, and the trace ends. Note that our methodology terminates on revisit, which may overstate loop frequency, but the underlying dynamic of getting stuck in circular navigation applies universally.

What causes loops? Unclear link hierarchy is the primary driver. Multiple links point to the same destination, so the agent thinks it is choosing between options, but they lead to the same page. Poor semantic navigation is another cause. Labels like "Platform" and "Solutions" are not semantically distinct for AI, even if they are for humans.

Broad information architecture with many paths to the same content also contributes. What looks flexible to humans confuses agents. Furthermore, some modern sites feature navigation where the homepage links back to itself, creating immediate loops.

This matters more than "page not findable" because loops indicate a navigability problem, not a discoverability problem. Content exists; the structure is confusing. The fix is different: clarify navigation, reduce link duplication, and audit link semantics. This is within your control, unlike search landing page selection.

To fix this, audit whether multiple navigation links go to the same page. Test whether you can reach target content from a given starting point without revisiting any page. Simplify the structure. One path is fine; ten paths to the same page is confusing. This compounds with depth: at 3+ clicks, loop probability is high because the agent has already visited several pages. Loops reduce when navigation is clear and labels are semantic.

Industry Variation: What Makes Sites Harder or Easier

Industries perform differently, even on the same tasks. Understanding these variations reveals what structural choices help or hinder AI navigation. Raw performance by industry shows SaaS and Financial Services at 84%, Content/Media and Enterprise/B2B at 79%, E-commerce at 74%, and Pharma/Healthcare at 68%.

However, headline numbers hide task difficulty confounding. When normalised by easy, medium, and hard task tiers, the variation tells a clearer story about information architecture.

Industry	Easy	Medium	Hard	Range
SaaS	83%	78%	55%	28pp
Financial Services	91%	73%	48%	43pp
E-commerce	80%	68%	48%	32pp
Enterprise/B2B	78%	58%	30%	48pp
Pharma/Healthcare	84%	50%	35%	49pp

SaaS performs best on hard tasks (55%). Modern product-led sites with clean information architecture and clear naming conventions pay off when tasks are complex. Product-focused design is robust against complexity.

Financial Services shows a strong baseline (91% on easy tasks) but a sharp cliff (48% on hard). Regulatory requirements enforce plain language and logical structure, which helps with standard tasks. But the structure fails on complex queries where semantic search matching is required.

Content/Media sites tie SaaS at 55% on hard tasks. Content sites optimise for browsing and discovery, creating a predictable information hierarchy that works for AI.

E-commerce is a stable performer. Built for discovery, it maintains consistency across tiers (80% easy, 68% medium, 48% hard). Design for "browsing" works reasonably well for AI navigation.

Enterprise/B2B shows the steepest collapse. From 78% on easy to 30% on hard is a 48pp drop. "Solutions-speak" navigation ("Platform," "Transform Your Business," "Capabilities") confuses semantic matching. Vague, jargon-heavy navigation breaks down at complexity.

Pharma/Healthcare collapses on medium and hard tasks (50% and 35%). Regulatory infrastructure creates interstitials, disclaimers, and geographic restrictions. These don't block browsing for humans but trip up agents.

The structural lesson is consistent: plain language beats jargon. Logical hierarchy beats flat or nonlinear organisation. Clear categories beat "solutions-speak." Some regulatory overhead is unavoidable, but it compounds with other issues. If you are in Enterprise/B2B, you have a harder problem. If you are in SaaS, your modern design choices are paying off.

What This Means for Your Site

These patterns are measurable on your site. You do not need to rely on speculation. You can audit your navigation against the findings to identify friction points.

Start by auditing your entry points. Test search-first navigation to see if a Google search lands on the right page. Test homepage navigation to see if you can reach key pages in 1-2 clicks. Compare where agents succeed versus where they fail to identify landing page issues.

Next, map your two-click zones. Identify which pages need to be reachable in 0-2 clicks. Pricing, contact, and key product pages should usually be 0-1 click from anywhere. Supporting content is reasonable at 1-2 clicks. Buried content at 3+ clicks is risky and prone to failure.

Audit navigation clarity. Does link text accurately describe where you will land? Do multiple links point to the same page, creating loop confusion? Is navigation language plain or jargon-heavy? Semantics matter.

Finally, check for loops. From your homepage, try to reach each key destination without revisiting any page. If you hit the same page twice, navigation is confusing.

These insights allow you to move beyond guesswork. AI agents don't care about your keywords or your backlinks. They care about whether they can find the answer.

Want to test how AI agents actually navigate your site? Compass simulates these patterns across your entire site—no manual testing required. See where your navigation succeeds and where it loops.