Last updated: August 21, 2026
AI search readiness is whether AI systems can reach your pages, read what they find, and name your company when a buyer asks. These three gates fail in sequence: a site that passes content quality but blocks crawler access never gets read. A site that is fully accessible but never gets recommended spent effort on table stakes. Fixing them in order is the whole game.
| Gate | The question | You pass when |
|---|---|---|
| Reach | Can AI crawlers access your website? | Major AI bots receive 200 responses and log no silent failures |
| Parse | Can AI read your pages? | Content renders without JavaScript and the accessibility tree passes audit |
| Recommend | Does ChatGPT name your company? | Your brand appears as a direct answer to buyer-phrased questions |
Most checklists on this topic stop after gate two and never test the part that actually drives revenue. This post walks through all three in order, because each gate is wasted effort without the one before it.
Gate 1, Reach: Can AI crawlers access your website?
AI crawler traffic now exceeds human traffic in volume. Automated bot requests surpassed human requests on Cloudflare’s network for the first time, and Cloudflare’s May 2026 data showed AI-related crawlers at roughly 20.3% of verified bot traffic, plus a further roughly 6.5% from AI-search bots, as measured by Cloudflare Radar. But not all of that traffic has anything to do with search visibility. Most of it is training-data harvesting, not answer-engine crawling that surfaces your pages to buyers.
Most AI crawl volume is training-data harvesting. Cloudflare’s crawler-purpose breakdown, measured between July and August 2025, found training accounted for roughly 80% of AI bot crawl volume, with “user action” and “undeclared” purposes combined under 5%. A much smaller slice is actual answer-engine crawling, the kind that feeds into ChatGPT, Perplexity, or Gemini when a buyer asks a question. The training crawls consume bandwidth and server resources, but they do not put your pages in front of a buyer. For teams tracking crawl volume, this means most of what hits the server is invisible to customers and irrelevant for visibility work.
Even when AI crawlers do visit with a search purpose, they take vastly more than they give back. Cloudflare’s analysis put ClaudeBot’s crawl-to-referral ratio at roughly 23,951 pages crawled per referral sent, versus Google’s traditional web-search ratio of about 4.9 to 1. Exact ratios move month to month per Cloudflare’s own reporting, but the asymmetry is consistent. Passing Gate 1 alone is not a growth channel. It is table stakes. A site can be wide open to AI crawlers and still receive almost no referral traffic if the content does not align with what the answer engines choose to surface. Opening access is the prerequisite, not the payoff.
The crawler landscape itself is shifting fast. Anthropic’s ClaudeBot posted the largest single-crawler surge on record, growing roughly 66% to become the #2 AI crawler by share (roughly 20.0%), passing Meta-ExternalAgent and GPTBot, while Googlebot’s share of AI-adjacent bot traffic dropped to a new low, as measured by Cloudflare. Which bots matter for your site changes month to month, and a robots.txt rule that was correct six months ago may now be blocking a crawler that feeds a major answer engine. Revisit your allowed-user-agent list at least quarterly.
Because Reach is a gate, not a growth lever, the work here is about removing accidental blocks. Start with robots.txt. A deeper walkthrough of which bots exist and how to audit your robots.txt against the four biggest AI crawlers, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, is in our bot-access guide. But robots.txt is not the only layer that can block access. A common blind spot is CDN or firewall bot management rules that block AI crawlers even when robots.txt permits them. The post on website blocking AI crawlers explains how to check for that layer of accidental blocking.
Once access is confirmed on paper, the next step is to see what is actually happening. To move beyond what robots.txt says is allowed and see what AI crawlers are doing on your server right now, the post on tracking AI crawler activity covers the tools and log patterns that make that visible. Many crawl attempts fail silently, returning errors or getting blocked at the network layer without anyone noticing. The piece on failed AI crawl attempts walks through how to surface those hidden failures. Even when access is allowed, broken paths and 404s can quietly remove pages from an AI crawler’s reach. The guide to fixing 404 errors for AI crawlers covers how to find and repair them.
Gate 2, Parse: Can AI read your pages?
Reaching the page is only half the battle. The AI must be able to read the content and understand the page structure. Many industrial and manufacturing sites rely on JavaScript to render key content. If the meaningful text and navigation elements only appear after client-side scripts execute, an AI crawler or agent may see an empty shell. Checking whether core pages render meaningful content without JavaScript is the first Parse check, and it requires comparing the raw page source to the rendered DOM, not just eyeballing the page in a browser.
Beyond content visibility, Chrome now measures whether a page is *operable* by an AI agent. Chrome’s Lighthouse added a dedicated Agentic Browsing audit category, evaluating four things: WebMCP tool registration via the Chrome DevTools Protocol, accessibility-tree integrity (names, labels, role hierarchy, visibility of interactive elements), layout stability via Cumulative Layout Shift, and presence of an llms.txt file at the domain root. It does not produce a 0-100 score like other Lighthouse categories. It reports a pass ratio, because the underlying standards are still emerging. The audit requires Chrome 150+ (or Chrome 130-149 with a developer flag enabled).
The accessibility tree is the structured representation of a page that screen readers and, increasingly, AI agents use to navigate and interact with content. If the tree is broken, missing labels, or hiding interactive elements, the agent cannot operate the page reliably. Layout stability matters for the same reason: a page that shifts under an agent mid-interaction can cause it to click the wrong element or abort the task. For a deeper walkthrough of what the audit checks and how to interpret the results, see our post on Chrome’s Agentic Browsing audit.
One piece of the Parse gate that gets outsized attention is llms.txt, a plain-text file placed at the domain root that lists key pages and context for LLMs. Adoption is real but still thin. A June 2026 review found roughly 8.7% of the top 1,000 sites publish an llms.txt file, and a separate study across 300,000 domains found roughly 10.13% adoption, holding roughly flat across low-, mid-, and high-traffic tiers. But llms.txt is a courtesy file, not a requirement. No formal standards body has adopted it as of April 2026; it remains a community convention maintained via llmstxt.org. Major LLM crawlers are not confirmed to fetch it, and SEO-citation studies so far show no measurable citation-rate improvement from having one. If you are deciding where to spend a limited afternoon, fixing broken accessibility labels will return more than publishing an llms.txt file.
Publishing an llms.txt file is low-cost hygiene. If it takes an hour to create and maintain, the downside is near zero. But treat it as a signal of organizational tidiness, not a growth lever. The llms.txt setup guide walks through the format and deployment steps if you decide to publish one.
Gate 3, Recommend: Does ChatGPT name your company?
The gap between being readable and being recommended is where revenue impact lives. A site can pass Reach and Parse perfectly and still never appear in an AI-generated answer. The question is not whether the AI *can* read your pages. It is whether the AI *chooses* to cite you when a buyer asks a real question.
B2B buyer behavior has shifted toward AI-driven research with a speed that makes this gate urgent. Forrester’s Buyers’ Journey Survey, 2025 found that 94% of B2B buyers used AI during their most recent purchase, up from 89% the year prior, a 5-point year-over-year increase. Twice as many B2B buyers now name generative AI or conversational search as their single most meaningful research source than any other channel, more than vendor websites, sales reps, or product experts, per the same survey. If your company is absent from AI answers, you are invisible to the fastest-growing research channel in B2B buying.
In a Semrush-commissioned survey of 622 B2B buyers fielded March-April 2026, 66% said they regularly use AI to research products or vendors for their job, and 92% said AI has shaped their vendor shortlist (45% said “significantly”). 69% reported using AI tools daily. The shortlist is the new battleground: being recommended by AI is not a brand-awareness metric. It is a pipeline metric.
Meanwhile, traditional search is leaking volume. U.S. Google searches ending without any click reached 68.01% in the January-April 2026 window, up from 60.45% in 2024, per a SparkToro analysis using Similarweb clickstream panel data. More searches resolve inside the search results page or inside an AI interface without ever sending a visitor to a website. Every zero-click search is a potential buyer who never sees your domain.
The traffic that does arrive from AI referrals behaves differently. Adobe Digital Insights found that AI-referred traffic to retail sites grew 138% year-over-year as of May 2026, part of a longer trend Adobe describes as roughly 14x growth since October 2024. Those numbers are retail-specific, so treat them as directional for B2B, not a direct measurement. But the engagement patterns are worth noting: AI-referred shoppers spend 53% more time on-site and view 23% more pages per visit than non-AI visitors, and by March 2026 AI-referred traffic converted 42% better than non-AI traffic, a reversal from converting roughly 38% worse a year earlier. For B2B teams, the signal is clear: AI-referred visitors arrive with more intent and engage more deeply.
Checking Gate 3 means running real buyer-phrased questions through ChatGPT, Perplexity, and Gemini and seeing whether your brand is named at all. When it is named, the distinction between a mention and a recommendation matters. A mention alongside five competitors is not the same as being the answer the AI puts forward. The post on checking AI visibility walks through how to run those queries and interpret whether the answer is a recommendation or just a mention.
This gate sits at the intersection of traditional SEO, answer engine optimization (AEO), and generative engine optimization (GEO). AEO is the practice of optimizing content to appear in AI-generated answers across platforms like ChatGPT, Perplexity, and Gemini. GEO extends that to include optimization for the retrieval and ranking mechanisms inside generative models specifically. For a breakdown of where the three-gate framework fits relative to those terms, see the comparison post on AEO vs GEO vs SEO.
For companies whose buyers are already asking AI for shortlists, the AI Visibility Audit measures which competitors the engines name and why, mapping the gap between where you appear and where you need to be.
The AI search readiness checklist
Here is the readiness checklist, gate by gate. Walk through it today, and set a re-check cadence.
Gate 1: Reach
- Check your robots.txt for explicit disallow rules aimed at the four biggest AI crawlers: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. The full list and how to read it are in the bot-access guide.
- Check CDN or firewall bot-management rules for AI-crawler blocking that is not visible in robots.txt.
- Pull server logs or a bot-analytics tool to confirm which AI crawlers have actually hit the site in the last 30 days.
- Check for 4xx and 5xx responses served specifically to AI user-agents. These are crawl failures that often go unnoticed.
- Confirm sitemap.xml is current and referenced in robots.txt.
Gate 2: Parse
- Confirm key pages render meaningful content without JavaScript. Compare page source to the rendered DOM.
- Run Lighthouse’s Agentic Browsing audit: note the accessibility-tree pass rate, CLS score, llms.txt presence, and WebMCP status.
- Check heading structure and answer-first formatting on the pages that matter most commercially.
- Confirm structured data (Organization, Article, FAQ, Product as relevant) validates.
- Decide on llms.txt. Publish one if it costs little, but track it as hygiene, not a growth lever.
Gate 3: Recommend
- Run real buyer-phrased questions through ChatGPT, Claude, Perplexity, and Gemini. Is your brand named at all?
- When you are named, are you recommended, or just mentioned alongside competitors?
- Identify which competitor consistently wins the recommendation and why.
- Check whether your brand entity is disambiguated correctly, not confused with a similarly named company.
- Set a re-check cadence. This gate moves fast and shifts month to month.
The free AI Agent Readiness Check covers Gates 1 and 2 mechanically. It checks your robots.txt, tests page rendering, runs the Lighthouse Agentic Browsing audit, and reports the results. Reading the framework tells you what to look for. The tool checks it for you.
What does “AI search readiness” actually mean?
It means passing three sequential gates. Reach: AI crawlers and agents can access your pages. Parse: they can read and operate the content. Recommend: the AI actually names your company when a buyer asks a real question. A site is not ready until all three are working.
Is AI search readiness the same thing as SEO?
No. Traditional SEO optimizes for search engine rankings and click-throughs. AI search readiness adds the layers of crawler access for AI-specific bots, agentic operability, and presence in AI-generated answers. The two disciplines overlap, but AI readiness addresses channels that do not always show up in organic search reports.
How do I know if ChatGPT or Perplexity actually recommend my company?
Run a set of buyer-phrased questions through each platform and record whether your brand appears in the answer. Distinguish between a mention alongside competitors and a direct recommendation. Repeat the check regularly, because answer patterns shift as models update and competitors publish new content.
How often should I re-check AI search readiness?
Crawler behavior, answer engine models, and buyer AI adoption are all still shifting. A quarterly re-check is a reasonable starting point. If your industry moves faster, or if you publish content frequently, a monthly cadence for Gate 3 recommendation checks makes sense. Treat readiness as a check-in, not a one-time project.
The sources AI assistants cite also move on their own. In August 2026, Reddit’s share of ChatGPT citations fell by 86 to 97 percent within days of a model update, a shift tracked under AI citation volatility.
Reading the framework tells you what to look for. The AI Agent Readiness Check checks it for you. We ran the check on our own site first and went from one passing check of three to three of three after the fixes.
