Last updated: August 21, 2026
To fix 404 errors for AI crawlers, locate the broken URLs by filtering your server logs for AI user agents and 404 status codes. Then assign each URL one of four responses: redirect to the closest real page, build the page the AI keeps inventing, return a 410 for permanently retired pages, or leave the 404 with a helpful error page. Never blanket-redirect to the homepage.
Broken URLs are one part of AI search readiness, the reach layer: whether AI systems can get to your content at all.
How to Find Which URLs AI Bots Are 404ing On
If you still need to assemble the list, the diagnosis step that comes before this post is covered in failed AI crawl attempts. Once you have the URLs, you can apply the fix framework below.
The most direct source is your server logs. A server log is the file where the web server records each request it received, including the requesting user agent, the identifying string a crawler sends, and the response status code. Filter for AI crawler user agents and a 404 status. If raw logs are not accessible, a CDN dashboard often shows the same bot request data by user agent and status.
Do not look for this in Google Search Console’s Crawl Stats report. Google Search Console’s Crawl Stats is scoped to Googlebot request and response data only. It does not report GPTBot, ClaudeBot, PerplexityBot, or any other AI crawler’s requests or status codes. That is a structural limitation of the tool, not a setting you can enable.
For hallucinated URLs that already produced a visit from ChatGPT, use GA4, the standard free web-analytics tool. In GA4, filter Engagement > Pages and Screens by “Session source/medium contains ChatGPT,” identify pages showing a 404-page title, then add “Page path and screen class” as a secondary dimension to see the exact hallucinated URLs.
Fix Strategies: When Each Applies
Every missing URL gets one of four responses. Redirect it to a real replacement, build the page, serve a 410, or let it 404 with a useful error page.
A 301 redirect is a permanent redirect from one URL to another. Google’s guidance on replacements is narrow. If there is no genuinely relevant replacement page for a dead URL, let it 404. When a real page used to exist at that URL and has moved or been renamed, a 301 to the new URL preserves the existing signal and backlink value.
The more interesting case is the URL the AI keeps inventing. A hallucinated URL that generates real referral traffic or already has inbound backlinks is a documented signal of content demand. One analysis found roughly 20% of hallucinated URLs it studied had at least one existing backlink pointing at them. There is no clean published threshold for when building the page is worth it. Backlink presence and repeat traffic are the best available signals, not a formula. If either signal is strong, the hallucination is functioning as content-gap research: build the page, ship it, and add it to the sitemap so it is discoverable beyond AI referrals.
For pages that were deliberately and permanently retired, serve a 410 rather than a plain 404. Both status codes tell the crawler the page is missing, but a 410 explicitly says the removal is permanent and intentional. Google wants to confirm a 404 before de-indexing through repeated checks over time. A 410 can prompt faster removal from the crawl queue because there is no ambiguity to resolve, though Mueller has described the practical ranking difference as small.
Finally, the 404 page itself is part of the fix. Practical Ecommerce’s stated sequence is to build a genuinely helpful 404 page first, then 301-redirect only the broken URLs generating measurable traffic, then keep monitoring. A good 404 page still serves navigation, search, and related links to any agent or human that lands there.
| Situation | Fix | Why |
|---|---|---|
| A real page used to exist at this URL, moved or renamed | 301 redirect to the new URL | Preserves the existing signal/backlink value; matches Google’s own guidance for genuine replacements |
| AI keeps hallucinating a URL with no real historical page, but it pattern-matches real site content | 301 redirect to the closest genuinely relevant existing page | Same logic as above, but only when a real match exists. Otherwise this becomes a soft-404 |
| AI keeps hallucinating a specific URL, and it’s generating real referral traffic or already has inbound backlinks | Build the actual page at that URL | The hallucination is free content-gap research; about 20% of hallucinated URLs in one study already had backlinks. Ship it, then add it to the sitemap so it’s discoverable beyond AI referrals |
| A page was deliberately, permanently retired (discontinued product, expired offer, old policy) | Serve a 410 rather than a plain 404 | Removes ambiguity for crawlers; marginally faster de-indexing; correct semantic signal even though the practical ranking difference is small |
| No real match exists and the URL isn’t worth building | Let it 404, with a genuinely useful 404 page | Google’s own advice: don’t redirect to the homepage; “make a better 404 page instead.” A good 404 page still serves navigation, search, and related links to any agent or human that lands there |
| Tempted to redirect all 404s to the homepage to “clean up” logs | Don’t | Google treats it as a soft 404 anyway. The original URL still drops from the index, you gain nothing, and you add user/agent confusion |
| Tempted to block the offending bot to stop the noise | Don’t, unless it’s a training-only crawler you’ve already decided to exclude for unrelated reasons | Blocking forecloses citation entirely; better to fix the crawlability of what does exist |
| Need to find which URLs are being hit before deciding | Filter server logs / CDN bot analytics by AI user-agent and 404 status; separately, filter GA4 by AI referral source and “404” page title | GSC Crawl Stats won’t show this; it’s Googlebot-only. GA4 method detailed in the how-to section above |
Use the table as a decision log. The two traps that cause the most self-inflicted damage are the last two rows: redirecting everything to the homepage and blocking the offending bot.
Why AI assistants hit 404s more than Google does
AI assistants are measurably more likely than Google’s crawler, Googlebot, to cite URLs that return a 404, the standard HTTP response code for “page not found.” In SE Ranking’s December 2025 analysis of 100,000 long-tail prompts, ChatGPT returned a 404 on 1.22% of all cited URLs, and 91% of those were true “404 Not Found” responses. Google AI Overviews returned a 404 on 0.56% of cited URLs in the same comparison.
The same pattern shows up independently in Ahrefs’ Brand Radar study, published September 2, 2025, based on 16 million unique URLs. Ahrefs found AI assistants send visitors to 404 pages 2.87 times more often than Google Search overall. These numbers come from different studies measuring different samples, but both point the same direction: AI-cited URLs fail at a multiple of Google’s baseline. Google’s John Mueller has also said hallucinated or broken AI-cited links might “increase in the short term” before AI systems become “better grounded in real URLs.” That is an acknowledgment from Google itself that this is a known, unsolved problem.
An AI crawler is an automated program that requests pages from a site on behalf of an AI product, such as GPTBot for OpenAI or ClaudeBot for Anthropic. To understand why those crawlers hit more dead ends, it helps to separate two measurements. A clicked-URL 404 rate measures how often a visitor or assistant follows a link and lands on a missing page. An all-cited URL 404 rate measures how often any URL the AI displays is broken, whether or not a human clicks it.
In Ahrefs’ clicked-URL data, ChatGPT’s 404 rate was 1.01%, compared with a 0.15% Google baseline. Claude was 0.58%, Copilot 0.34%, Perplexity 0.31%, Gemini 0.21%, and Mistral 0.12%. On all cited URLs, ChatGPT’s rate was 2.38%, compared with 0.84% for Google’s own SERP. Perplexity was at 0.87%, Gemini 0.86%, and Copilot 0.54%.
Ahrefs interprets that gap this way: Perplexity and Gemini sit close to Google’s baseline, which suggests those two assistants rely more on indexed or retrieved sources. ChatGPT’s much higher rate points to more URL generation from trained memory rather than retrieval. That is Ahrefs’ own interpretation, not an independently confirmed mechanism.
The problem is visible in server logs as well as in aggregate studies. On a single monitored e-commerce site, Oncrawl recorded ChatGPT-User generating 988 requests resulting in 404s, concentrated over several hours, hallucinating non-existent product URLs. This is not a rare crawler behavior at the margins. GPTBot and ClaudeBot accounted for roughly 12% and 9.2% of global AI-bot-classified traffic respectively from March 2025 through March 2026, versus 48% for Googlebot and 9.2% for Bingbot. GPTBot was also the single most-blocked bot in the world in 2025, more blocked than Googlebot or Bingbot, according to Cloudflare data cited by Oncrawl.
Some 404 traffic is also just an agent probing conventional paths. AI agents sometimes request guessed locations like /llms.txt, a community-proposed convention rather than an adopted access-control standard for any major AI search product. A missing file there is expected.
What Not to Do
Do not blanket-redirect every 404 to the homepage. Google’s John Mueller has said that redirecting dead pages to the homepage is treated as a soft 404 by Google regardless. A soft 404 is a URL that appears to exist because it redirects or returns a page, but Google treats it as missing anyway. The original URL still gets dropped from the index as if it had 404’d, so there is no ranking upside and it only adds user confusion. His direct quote is: “not a great practice (confuses users), and we mostly treat them as 404s anyway (they’re soft-404s), so there’s no upside… make a better 404 page instead.”
Do not block the bots just to stop the noise. Blocking a crawler like GPTBot removes your site from that assistant’s retrieval and citation pool entirely. It does not merely stop 404 requests. It forecloses any future citation. The more nuanced, frequently recommended approach is to block training-only crawlers while allowing the separate search-retrieval crawlers, such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot, that actually drive citations and traffic. For a deeper walkthrough of the block/allow decision, read Website Blocking AI Crawlers.
Some of the 404 noise is probing for conventional paths. AI agents sometimes request /llms.txt, a community-proposed convention. It is not an adopted access-control standard for any major AI search product, so a missing file there is expected and not a 404 worth chasing. Whether to set one up is a separate setup question, covered in the llms.txt setup guide, not a 404-fix tactic.
Does Fixing 404s Actually Improve AI Visibility?
The honest answer is that the causal case is not closed. No dedicated study in the sources reviewed here quantified whether fixing 404s specifically, as opposed to overall crawlability or content quality, measurably improves how often an AI assistant cites your site.
What is verified is that AI assistants are measurably more error-prone than Google at generating working links. SE Ranking found ChatGPT’s cited-URL 404 rate at 1.22% versus 0.56% for Google AI Overviews. Ahrefs found AI assistants send visitors to 404 pages 2.87 times more often than Google Search overall.
The logic here is that crawl budget, the attention a crawler gives your site, is finite. Every request that ends in a 404 is time a bot is not spending on a real page. This follows from the mechanics of crawling, but it is a plausible mechanism, not a proven outcome.
Keep the scale in view. Across monitored sites in the Oncrawl dataset, AI-crawler traffic ranged from 0.1% to 1% of total site traffic, with a median of about 0.4%. That is bot-crawl traffic, not AI-referred human visitor traffic. Fixing 404s is unlikely to swing overall traffic dramatically. It removes a known failure mode in how AI systems and human visitors move through the site. Treat it as infrastructure hygiene, not a guaranteed citation boost.
FAQ
Do 404s hurt my visibility in ChatGPT or AI Overviews?
No direct penalty is established. The verified pattern is that AI assistants, especially ChatGPT, generate broken citations more often than Google does. Fixing 404s removes a known failure point, but the citation impact is not a measured outcome.
Why does ChatGPT link to pages that don’t exist on my site?
Ahrefs’ interpretation is that ChatGPT generates more URLs from trained memory rather than looking them up in a live index. That is Ahrefs’ own interpretation, not a confirmed mechanism. Real-world logs support the scale: one monitored e-commerce site saw ChatGPT-User produce 988 404 requests over several hours, all hallucinated non-existent product URLs.
Should I redirect every broken AI-bot request to my homepage?
No. Google’s John Mueller says redirecting dead pages to the homepage is treated as a soft 404. The original URL still drops from the index, there is no ranking upside, and it adds confusion. Let the URL 404 unless a genuinely relevant replacement page exists.
What’s the difference between a 404 and a 410, and does it matter for AI crawlers?
Both tell a crawler the page is missing. A 410 adds that the removal is permanent and intentional, which can prompt faster removal from the crawl queue because Google does not need repeated checks to confirm it. The practical ranking difference is small.
Can I see which URLs AI bots are 404ing on in Google Search Console?
No. Google Search Console’s Crawl Stats report is scoped to Googlebot only. It does not report requests or status codes from GPTBot, ClaudeBot, PerplexityBot, or other AI crawlers. Use server logs, CDN bot analytics, or GA4 instead.
The 404s you just found are the visible failures. The invisible ones are on pages that load fine but that an AI agent cannot read or operate. The free AI Agent Readiness Check tests that layer.
