Can AI Access Your Site?

Why AI Access Matters

AI agents now drive a meaningful share of discovery decisions. ChatGPT, Claude, Perplexity, Gemini, and Copilot answer questions by reading live web pages or by retrieving content from indexes built by crawlers. If those agents cannot reach your site, your brand does not exist in those answers. This is not a future problem: research shows that AI referral traffic converts at significantly higher rates than traditional search, which means the small volume of clicks that do arrive are disproportionately valuable.

The first technical gate is access. Before an AI can cite your pricing, summarise your product, or recommend your service, it must be able to fetch the page. Two failure modes are common. First, the site blocks the AI user-agent at the HTTP layer, returning 403 Forbidden or similar errors. Second, the site blocks the crawler via robots.txt, which prevents indexing and training even when a direct fetch might succeed. Our AI Access Checker tests both failure modes in one scan.

What This Tool Tests

The checker runs two independent tests against any URL you provide. The first is a live HTTP access test. We send real requests using the same user-agent strings that OpenAI, Anthropic, Google, Perplexity, and other platforms publish. We group these into three tiers: user-initiated browsing, AI search indexing, and training or crawl access. Each tier has different implications for visibility.

The second test fetches and parses your robots.txt file. This file is the published policy that tells compliant crawlers which parts of a site they may visit. Most AI platforms respect robots.txt, which means an overly broad Disallow directive can remove you from AI search results entirely. We compare your robots.txt against 18 known AI agent identifiers and report which ones are blocked.

Understanding robots.txt for AI

robots.txt is a simple text file, but its consequences are large. A single line can block every AI crawler from reading your content. The most common mistake is a wildcard disallow such as User-agent: * followed by Disallow: /, which blocks all bots including AI. Another frequent error is copying a legacy SEO robots.txt that targets specific crawlers without realising it also catches newer AI agents.

The default recommendation for most sites is to allow access. Unless you are protecting proprietary data, paywalled research, or content you do not want used in model training, blocking AI crawlers reduces visibility without providing meaningful protection. Even if you do not care about being indexed for AI search, consider that user-initiated browsing from ChatGPT or Claude may still attempt to fetch your pages in real time.

Common Access Mistakes

• Blocking every bot by default. A wildcard disallow is easy to add and hard to notice. It removes you from search and AI simultaneously.
• Relying on IP allow-lists. AI crawlers operate from cloud IP ranges. If your firewall or WAF blocks non-human traffic aggressively, AI agents may be rejected before robots.txt is even checked.
• Hiding content behind JavaScript. Even when access is allowed, content loaded by client-side JavaScript may be invisible to lightweight AI fetchers. This is a rendering problem, not an access problem, but the result is the same: AI cannot see your content.
• Forgetting subdomains and staging sites. Your main domain may allow AI while a subdomain, documentation site, or help centre blocks it. Each property needs its own check.

How to Fix Access Issues

If the tool reports blocks, start with robots.txt. Remove unnecessary disallows for AI user-agents, or replace a broad block with targeted rules that protect only sensitive paths. A safe default looks like this:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://www.yoursite.com/sitemap.xml

If HTTP access is blocked but robots.txt is fine, investigate your server, CDN, or WAF configuration. Look for bot management rules that reject requests based on user-agent, IP reputation, or request signatures. You may need to explicitly allow the user-agents listed in our results.

What the Results Mean

A green result means the agent can fetch the page and is not blocked by robots.txt. A red result means either the HTTP request failed or the robots.txt policy disallows the agent. An amber result usually indicates a transient error, such as a timeout or an unusual redirect, which may need manual investigation.

Remember that access is necessary but not sufficient. An AI agent that can fetch your page may still fail to navigate your site, understand your content, or cite you accurately. That is why access testing is the first step in a broader AI readiness programme.

Who Should Use This Tool

This checker is useful for anyone responsible for digital visibility. Marketing teams can verify that new campaigns are not accidentally blocked. SEO teams can confirm robots.txt policies align with AI strategy. Product and engineering teams can debug access issues before they affect referral traffic. Agencies can run quick checks for clients during onboarding or after site migrations.

If you manage multiple properties—main site, blog, help centre, documentation—run the check for each domain and subdomain. A single robots.txt file at the root does not govern subdomains. Staging environments also need checking; it is common for a staging block to be copied to production accidentally.

Limitations of This Check

This tool tests policy and basic HTTP access, not rendering quality or navigation success. A page can be accessible but still invisible if its content loads via JavaScript that AI agents do not execute. It can be accessible but unreachable if your site structure buries it three clicks deep. It can be accessible but uncitable if your content is ambiguous or poorly structured.

robots.txt compliance is also voluntary. Most major AI platforms respect it, but bad actors and some specialised crawlers may not. A green result here does not guarantee protection from scraping, nor does a red result guarantee that no AI platform will ever see your content. Use this tool as one input in a broader AI readiness programme.

How Often to Re-check

Check access after any change that touches infrastructure, DNS, or crawler policy. That includes robots.txt updates, CDN migrations, WAF rule changes, hosting provider switches, and major platform upgrades. Set a calendar reminder to re-check quarterly even if nothing has changed, because platform policies and your own configuration can drift.

For larger sites, consider adding robots.txt parsing to your continuous deployment pipeline. A simple test that fails the build if critical AI agents are blocked can prevent accidental lockouts that would otherwise go unnoticed for weeks.

Interpreting Edge Cases

Some results require context. A site that returns 403 for some AI agents but 200 for others usually has bot-management rules that discriminate by user-agent. A site with no robots.txt is not automatically blocked, but it lacks a clear policy. A site that allows all crawlers but serves empty HTML has an access problem disguised as a rendering problem.

Subdomains and path-level blocks matter too. A root domain might allow ChatGPT while a documentation subdomain blocks it. Always test the specific URLs you care about, not just the homepage. Path-level robots.txt rules can allow most of a site while blocking a critical section like /blog or /products.

Sharing and Acting on Results

Use the results to start a conversation with the right team. A robots.txt block is usually a marketing or engineering decision. An HTTP block often points to WAF, CDN, or hosting configuration. Share the specific user-agents and statuses with your engineering team so they can reproduce the issue.

Track results over time by recording the date, URL tested, blocked agents, and any actions taken. This creates an audit trail and helps you demonstrate progress as you improve AI accessibility across your properties.

Next Steps

Once your site is accessible, the next question is whether AI agents can actually use it. Can they find your pricing page from the homepage? Can they understand your product descriptions? Do they get stuck in navigation loops? These questions require a deeper audit of navigation, content extraction, and site structure.

For a complete picture, run a Compass audit. Compass simulates real AI navigation across your site, identifies rendering blind spots, and gives you a prioritised action plan. The AI Access Checker tells you whether the door is open; Compass tells you whether AI can find what it needs once it walks in.