How to Audit a Website for AI Search Visibility

Most “AI visibility audits” you’ll find online are a checklist wearing a lab coat. Add schema. Write FAQs. Ship an llms.txt. Check your robots.txt. Done.

The problem is that a checklist has no order of operations, and AI search absolutely does. A model has to reach your page, read it, retrieve it, choose to quote it, and then send someone back to you — in that sequence. Fail gate one and nothing downstream matters. Pass gates one through three and fail gate four, and you’ll sit in dashboards looking “crawled” and “indexed” while never appearing in a single answer.

That last failure is more common than the industry admits. Analysis of Ahrefs’ AI search research found that 85% of the pages ChatGPT retrieves never appear in the final answer (Quattr’s breakdown of the Ahrefs study, 2026). Retrieval is not citation. Most audits stop at the wrong gate.

This guide restructures the audit around the pipeline a machine actually walks. Five gates, each with a pass/fail test you can run today, the tool that runs it, and the fix when it fails.

What Is an AI Search Visibility Audit — and How Is It Different From an SEO Audit?

An AI search visibility audit evaluates whether large language model–powered surfaces — Google AI Overviews and AI Mode, ChatGPT, Perplexity, Gemini, Copilot, Claude — can access, parse, retrieve, and cite your content when someone asks a question your business should answer.

A traditional SEO audit asks: can this page rank for a keyword? An AI visibility audit asks: can this page be quoted in an answer to a prompt?

Those are different questions with different failure modes:

Dimension

They overlap. They are not the same audit, and running only the first one is how sites end up dominant in Google and invisible in ChatGPT.

Why Can’t You Just Rely on Your Google Rankings?

Because the correlation has collapsed, and there is now hard data on it.

Ahrefs ran 15,000 long-tail queries through Google and Bing, then put the same questions to ChatGPT, Gemini, Copilot, and Perplexity. On average, only about 11% of the URLs cited by AI assistants also appeared in Google’s or Bing’s top 10 for the same query (Ahrefs, 2026). Perplexity was the most SEO-aligned of the four; ChatGPT and Gemini showed the least crossover.

Inside Google’s own AI features the alignment is stronger but weakening — the overlap between AI Overview citations and the top 10 organic results fell from roughly 76% to 38% in under a year (Quattr, 2026).

Meanwhile the click economics changed underneath everyone. SparkToro’s 2026 study found 68% of US Google searches now end without a click, with the share of searches producing at least one click falling 9.51 percentage points between 2024 and 2026 (Search Engine Land, June 2026). Similarweb’s 2026 Generative AI Landscape report put AI Overviews on 43% of Google searches, up from 15% a year earlier (DigitalApplied summary of the Similarweb/TechCrunch data, July 2026).

Two implications for how you audit:

1. Your rank tracker is not an AI visibility tracker. Position 3 tells you almost nothing about whether you’ll be quoted.
2. Traffic loss and visibility loss are now separate diagnoses. You can lose clicks while gaining citations, or hold clicks while disappearing from answers. An audit that only reads GSC can’t tell those apart.

Gate 1: Can AI Crawlers Actually Reach Your Site?

Start here because it is the cheapest failure to find and the most expensive to leave in place.

The mistake almost everyone makes is treating “AI bots” as one category. They are two, and they are independently controllable:

The strategic point: you can allow retrieval bots while disallowing training bots. Most teams have never made that choice deliberately — they inherited a robots.txt from a template or a WAF rule from a security vendor.

Run these four tests:

1. robots.txt review. Fetch yourdomain.com/robots.txt and check every user-agent above by name. Look specifically for a blanket Disallow: / under a wildcard that catches AI agents.
2. WAF / CDN rule audit. Cloudflare, Akamai, and AWS WAF now ship AI-bot-blocking presets that are sometimes on by default. This is the single most common silent failure in a Gate 1 audit — robots.txt says yes and the edge says 403.
3. Server log analysis. Pull 30 days of logs and count hits by user-agent. Zero GPTBot or OAI-SearchBot hits on a site with normal Googlebot traffic is a red flag, not a coincidence. For scale reference, one 30-day server-log study measured GPTBot at roughly 4,200 hits per site per day, with ClaudeBot near 1,800 and PerplexityBot near 980.
4. Response-code spot check. Run curl -I -A “OAI-SearchBot” https://yourdomain.com/key-page/ on your five highest-value pages. Anything that isn’t a 200 is a finding.

Common fixes: explicitly allow retrieval agents in robots.txt; whitelist their published IP ranges at the WAF; remove bot-challenge pages (CAPTCHA, JS interstitials) from money pages; confirm your CDN isn’t serving a different cached response to non-browser user-agents.

Gate 2: Can They Extract Meaning Without Running JavaScript?

This is where modern, well-built websites fail hardest — and where the audit finding is most likely to shock a dev team.

Vercel and MERJ tracked over 500 million GPTBot fetches and found zero evidence of JavaScript execution. GPTBot downloads JS files about 11.5% of the time and never runs them. The same held for ClaudeBot, PerplexityBot, Meta’s ExternalAgent, and Bytespider (summarized by Radiant Elephant, 2026).

Compounding it: roughly 92% of ChatGPT’s web-search responses draw on Bing’s index, and Bingbot’s JavaScript rendering is limited. A client-side-rendered React, Vue, or Angular app loses ChatGPT visibility from two directions at once — the direct crawl sees an empty shell, and the index it falls back on may have seen one too.

Run these five tests:

1. The view-source test. Open your page, Ctrl+U for view-source (not DevTools Elements, which shows the post-JavaScript DOM), and search for a full sentence of your main body copy. If it’s not there, no AI crawler has ever read it.
2. The curl test. curl -A “GPTBot” https://yourdomain.com/page/ | grep “your key sentence”. This is literally what the crawler gets.
3. JavaScript-disabled walkthrough. Disable JS in DevTools and navigate your top 10 pages. Loading spinners, blank sections, and empty pricing tables are all findings.
4. Structured data in the raw response. JSON-LD injected by a tag manager after page load disappears for AI crawlers. Your Organization, Article, FAQPage, Product, and LocalBusiness markup must be server-rendered. Validate with Schema.org’s validator against the raw HTML, not the rendered page.
5. Hidden-content audit. Accordions, tabs, “read more” toggles, and modal-loaded specs frequently hide the exact content most worth citing.

Common fixes: move to SSR or static generation for content pages (hybrid is fine — server-render the text, hydrate the widgets); inline critical JSON-LD server-side; convert tabbed content to visible sections with real headings; render pricing, specs, and FAQs into initial HTML.

Also check semantic structure while you’re here. AI systems chunk pages before they retrieve them. A single <h1>, a logical <h2>/<h3> hierarchy, real <table> markup instead of styled divs, <ul> for lists, and descriptive anchor text all make chunk boundaries cleaner. A wall of <div>s chunks badly, and a badly chunked page loses at Gate 4.

Gate 3: Are You Being Retrieved for the Prompts That Matter?

Keywords are not prompts. Auditing Gate 3 means building a prompt set and testing it systematically.

Build the prompt set (aim for 40–60 prompts across five types):

• Category discovery: “best [category] for [use case]”
• Comparison: “[your brand] vs [competitor]”
• Problem-first: “how do I fix [problem your product solves]”
• Qualification: “what should I look for when choosing a [category] provider”
• Branded: “is [your brand] any good”, “what does [your brand] do”

Weight them toward how people actually type into a chatbot — long, conversational, context-loaded. Ahrefs’ Q1 2026 benchmark found AI Overviews appear on just 9.5% of single-word queries but 46.4% of queries with seven or more words (Ahrefs Q1 2026 AI Search Benchmark Report). Short head terms are the wrong test set.

Run each prompt across every surface, logging four things:

Run each prompt at least three times — answers are non-deterministic. Use a fresh session or logged-out browser so personalization doesn’t contaminate results. Then calculate share of voice: your mentions divided by total brand mentions across the prompt set.

Also audit query fan-out. AI Mode and AI Overviews expand one question into many sub-queries and pull sources for each. That means ranking across a topic cluster beats ranking once for a head term. If your coverage is one strong pillar page with no supporting depth, you’ll lose the sub-queries you never thought to target.

Tools: Ahrefs Brand Radar, Profound, Otterly.ai, Peec AI, Semrush AI Toolkit — or a scripted approach with the platform APIs if you want full control over the prompt set and logging.

Gate 4: Why Do AI Engines Retrieve Your Page but Never Cite It?

This is the gate nobody audits, and it’s where 85% of ChatGPT-retrieved pages die (Quattr, 2026).

Retrieval means the system fetched your page as a candidate. Citation means it decided your passage was the best available answer. The gap between those two is an extraction problem, not an authority problem.

Audit at the passage level, not the page level. For each high-value page, take your top 10 target prompts and ask: is there a self-contained 40–90 word passage on this page that answers this question completely, without requiring the surrounding paragraphs for context?

If a model would have to stitch together three paragraphs to build an answer, it will usually pick a source where it doesn’t have to.

The Gate 4 checklist:

• Direct-answer openers. Does each H2 question get answered in the first two sentences beneath it, before the elaboration?
• Self-containment. Do passages avoid orphan pronouns (“this approach”, “as mentioned above”) that break when extracted?
• Specificity. Named numbers, dates, prices, and thresholds get quoted. “Significantly faster” doesn’t; “43% faster on average” does.
• Attributable claims. Original data, first-party research, and cited statistics are far more quotable than opinion.
• Freshness signals. Content recency matters more in AI search than in classic SEO. ChatGPT shows the strongest recency bias of the major engines — reporting on Ahrefs’ data found a large majority of its most-cited pages had been updated within the previous 30 days. Visible datePublished and dateModified in schema, plus a real “last updated” line, are cheap wins.
• Format match. Comparison prompts want tables. Process prompts want numbered steps. Definitional prompts want a one-sentence definition. Audit whether your format matches the prompt type you’re targeting.
• Contradiction check. If your pricing page, your FAQ, and your blog give three different answers to the same question, models resolve the conflict by citing someone else.

A useful diagnostic: paste your page into a model and ask it to answer your target prompt using only that page. If it struggles, hedges, or pulls from general knowledge instead of your text, your extractability is the problem — not your Domain Rating.

Gate 5: Is Your Off-Site Footprint Strong Enough to Be Recommended?

You can pass Gates 1–4 perfectly and still lose, because the strongest AI visibility signals aren’t on your website at all.

Ahrefs analyzed 75,000 brands and measured which factors correlate with appearing in Google AI Overviews. Branded web mentions correlated at 0.664. Backlinks came in at 0.218 — roughly three times weaker. The top three factors were all off-site brand signals: brand web mentions (0.664), branded anchors (0.527), and branded search volume (0.392) (Ahrefs, AI Overview brand correlation study).

The December follow-up extended the analysis across ChatGPT, Google AI Mode, and AI Overviews and found an even stronger signal: YouTube mentions correlated at approximately 0.737 — the highest of any factor tested, consistent across all three platforms (Ahrefs; TNW coverage). A plausible mechanism: YouTube transcripts put brand names into plain, machine-readable text at enormous scale.

Ahrefs also found brands in the top quartile for web mentions earned up to 10x more AI Overview mentions than the next tier down.

Audit your off-site footprint on five dimensions:

1. Unlinked mention volume and trend. Track mentions of your brand name across the web, linked or not. Ahrefs Brand Radar, Brand24, and Mention all do this.
2. Source independence. Owned content, syndicated press releases, and self-authored guest posts count for very little. Independent editorial coverage carries the weight.
3. Entity consistency. Your business name, address, category, and founding details should match exactly across your site, Wikipedia/Wikidata (if applicable), Crunchbase, LinkedIn, Google Business Profile, and industry directories. Inconsistency makes you a fuzzy entity, and fuzzy entities don’t get named confidently.
4. Community presence. Reddit, Quora, Stack Overflow, and niche forums are disproportionately represented in AI citation sets. Audit whether your category conversations mention you at all.
5. Video and multimedia. Given the YouTube correlation, a total absence from video is now a measurable visibility gap, not just a missed channel.

How Do You Measure AI Search Traffic That GA4 Hides?

By default, GA4 buries AI referrals in “Referral” and “Direct,” which is why most teams believe they get no AI traffic. They get some. They just can’t see it.

Set up a custom channel group:

Admin → Data display → Channel groups → Create new. Add a channel named “AI Search” with a Source match on:

chatgpt.com, chat.openai.com, openai.com, perplexity.ai, gemini.google.com,

copilot.microsoft.com, bing.com/chat, claude.ai, you.com, poe.com

Backfill by segmenting historical Referral traffic against the same list.

Then audit what the numbers mean. AI referral volume is small but the quality is not. Semrush’s research found AI-referred visitors convert at roughly 4.4x the rate of traditional organic visitors — a finding corroborated in direction, if not magnitude, across multiple independent studies. Conductor’s benchmark across 13,770 domains put AI referral traffic at about 1.08% of total sessions, while year-over-year growth ran in the hundreds of percent (data roundup, Omnibound).

Interpret that carefully in your audit: a 1% channel converting at 4x is worth about 4% of a 100% channel’s output — today. The case for investment is trajectory, not current volume. Say that plainly to stakeholders rather than overselling it.

Also account for dark traffic. A large share of AI influence never shows a referrer: users read an answer, then search your brand name and arrive via “organic branded” or type your URL directly. Two proxies worth tracking in your audit baseline:

• Branded search volume trend in GSC (rising branded impressions with flat non-branded is an AI-influence signal)
• Direct traffic to deep pages that no one would ever type from memory

What About llms.txt — Does It Actually Do Anything?

Short answer: not for search visibility. Include it in the audit, but score it honestly.

Google’s Search Central AI optimization guidance, updated June 2026, states that site owners don’t need to create machine-readable files, AI text files, or Markdown to appear in Google Search including its generative features — because Search doesn’t use them. John Mueller has compared llms.txt to the discredited keywords meta tag; Gary Illyes confirmed at Search Central Live that Google isn’t pursuing it.

The empirical picture matches. Limy analyzed over 500 million AI bot traffic events across a 90-day window and found only 408 targeted fetches of /llms.txt. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip the file and crawl HTML directly (Limy, May 2026).

How to score it in an audit: llms.txt is a coding-agent and agentic-browsing asset, not an AI search ranking asset. If you sell to developers or run a documentation-heavy product, ship one — Cursor, Copilot, Windsurf, and MCP-based tools do consume it. If you’re a local service business or an ecommerce brand, it is not a finding worth prioritizing, and any auditor who leads with it is selling you the easy deliverable instead of the hard one.

How Often Should You Re-Run an AI Visibility Audit?

Full audit: quarterly. Gate 3 monitoring: continuous. Anything less than weekly retrieval tracking and you’re measuring noise.

Conclusion: Audit the Pipeline, Not the Panic

There’s a version of this work that’s mostly theater — a 60-slide deck, an llms.txt, some FAQ schema, and an invoice. It’s popular because it’s fast and because clients can’t easily tell the difference between motion and progress.

The five-gate model exists to make that difference visible. Every finding has to name the gate it failed at, and every gate has a test that either passes or doesn’t. When a client asks why they aren’t showing up in ChatGPT, “we need to do GEO” is not an answer. “Your pricing page is client-side rendered and OAI-SearchBot has been receiving an empty div for eleven months” is.

The uncomfortable part of the data is that Gate 5 — the one you control least — carries the most weight. Branded web mentions out-predicting backlinks three to one means the highest-leverage work is often not on your website. That’s a hard sell for an audit deliverable, because the fix isn’t a ticket. It’s a year of being genuinely worth talking about, in places you don’t own.

So run the audit in order. Fix Gates 1 and 2 this month, because they’re binary and cheap. Instrument Gate 3 so you have trend data before you need it. Rewrite for Gate 4 as pages come up for refresh. And treat Gate 5 as what it actually is: not an SEO task, but the long, unglamorous work of building a brand that machines have heard of.

The engines changed. The reason you get recommended didn’t.