AI Assistants Invent URLs for My Site. I Logged Every 404.

TL;DR

What: AI assistants link people to pages on my site that never existed, so I logged every 404 that an AI bot hits. Why: an AI 404 is a silent failure, the user blames my site and never tells me. How: a Laravel exception-handler hook plus a user-agent whitelist, one database write per hit, no log parsing and no extra infrastructure. Proof, up front: 235 invented URLs in 26 days across 104 distinct paths, about 78% of them from OpenAI's bots, and several of the invented pages turned out to be worth building.

The problem nobody files a bug for

When you have a broken checkout, someone eventually complains. When an AI assistant invents a URL for your site, nobody does. The model writes a confident answer, drops in a clean-looking link to your domain, and the person clicks it and lands on a 404. They do not email you. They just close the tab and assume your site is the problem.

I run a fleet-tracking SaaS, and a growing share of its discovery now happens inside ChatGPT, Perplexity, and Claude rather than Google. Those tools cite URLs. The trouble is they sometimes cite URLs that I never created. I had no idea how often until I measured it, because the failure mode is invisible from my side: no error, no report, just a real person bouncing off a page that was never there.

So I decided to turn the invisible into a number.

How I logged AI-bot 404s without parsing logs

The capture is automatic. The response, once a week, is yours.

The obvious way to find these is to grep the access logs for 404s with AI user-agents. I did not want to. That means standing up log shipping, parsing, and storage, and babysitting it forever, to answer one question.

The cheaper path is to let the application tell me. When a request exhausts every route and throws a 404, the framework already knows. I just hook that moment:

// bootstrap/app.php
$exceptions->renderable(function (NotFoundHttpException $e, $request) {
    $bot = AiBots::match($request->userAgent());   // null if not an AI bot
    if ($bot) {
        Ai404Event::create([
            'path' => $request->path(),
            'user_agent_bot' => $bot,
            'occurred_at' => now(),
        ]);
    }
    return null; // let the normal 404 page still render
});

Two details made this worth writing down. First, it has to be renderable, not reportable. Laravel keeps NotFoundHttpException on an internal "do not report" list, so a reportable hook never fires for 404s. Second, it costs almost nothing: it runs only after routing has fully failed, only when the user-agent matches my list, and it does a single indexed insert. No worker, no log pipeline, no Cloudflare add-on.

The whitelist is just the AI crawlers and assistants worth tracking, matched as a case-insensitive substring of the user-agent:

GPTBot · ChatGPT-User · OAI-SearchBot · ClaudeBot · claude-web
Google-Extended · PerplexityBot · Perplexity-User

That is the whole machine. Every AI 404 now lands in one table I can rank.

What the AIs actually invented

Twenty-six days later, the table had 235 events across 104 distinct paths. The split by bot was lopsided:

The same split, by vendor: OpenAI dwarfs the rest.

OpenAI's three bots alone are 183 of 235, about 78% of the invented traffic. If you are going to keep one AI source honest, that is the one.

The invented paths fall into clear patterns:

• Use-case pages it assumes must exist. The single most-hit invented URL was gps-for/high-value-goods (31 hits), followed by gps-for/frigorifique, gps-for/trailer, and gps-for/bus. The AI reasoned that a tracking company surely has a page for each cargo type, and linked to it before checking.
• Product URLs with fabricated IDs. It invented store/toronto/product/bouncie-gps-tracker-... and product/tracki-pro-..., complete with plausible-looking unique ID suffixes that match my URL shape but point at nothing.
• Namespace confusion. It kept linking /blog/<slug> when my posts live at /post/<slug>.
• Dropped suffixes and doubled slugs. It shortened /alternatives/samsara-alternatives to /alternatives/samsara, and once produced trackers/wialon/wialon-wialon-retranslator.

Plausible shape, fabricated tail.

None of these are in my sitemap. The models generated them from the shape of my site plus what they expected a company like mine to have.

The false positives (and a real bug it caught)

Two findings kept me from over-trusting the data, and they are the interesting part.

First, actuator/env showed up five times wearing an AI user-agent. That is a Spring Boot secrets-leak probe, not anything an AI assistant would cite. It is almost certainly a vulnerability scanner spoofing a bot string. A user-agent whitelist is a starting point, not gospel, it will catch impersonators along with the real thing.

Second, and more useful: alternatives/samsara-alternatives was 404ing too. That is not a hallucination. That is the real, canonical comparison URL failing. My own redirect rule, the one meant to fix the dropped-suffix case, was shadowing the page it was supposed to protect. The AI-404 log accidentally surfaced a genuine bug in my own routing. I would not have found it otherwise.

From a 404 to a redirect

The capture is automatic, the response is not, and on purpose. Every week the events get ranked by frequency and land in my ops digest. I read them and decide what each one means, because only a human knows whether an invented URL deserves a redirect, a new page, or nothing.

When a pattern is obviously a naming mismatch, I write a redirect by hand. The real entries in my redirect file still carry the note-to-self that prompted them:

// AI assistants consistently hallucinate /blog/<slug> URLs
// but we use /post/<slug>. 6 distinct URLs hit in the last 30 days.
Route::redirect('/blog/{path}', '/post/{path}', 301)->where('path', '.*');

// AI drops the -alternatives suffix on competitor comparison pages.
// /alternatives/samsara, /alternatives/wialon

That is the boring half of the loop, and it works.

FAQ

Do AI assistants make up URLs? Yes, regularly. They generate links from the shape of your site plus what they expect a company like yours to have, then cite them before checking. In 26 days mine logged 235 invented URLs across 104 paths.

Which AI bot hallucinates the most? For my site, OpenAI's bots (ChatGPT-User, OAI-SearchBot, GPTBot) were about 78% of the invented hits, well ahead of Perplexity and Claude.

How do you catch them without parsing logs? Hook the framework's 404 handler, check the user-agent against a list of AI bots, and record a row only when both match. One database write per hit, no log pipeline.

From bug surface to demand signal

The interesting half is the reframe. I started this to stop AI assistants from dumping my visitors onto dead pages. What I ended up with was a ranked list of what AI thinks my product should offer.

gps-for/high-value-goods. gps-for/frigorifique. gps-for/trailer. Those are not just 404s to redirect. They are demand, surfaced for free, by the tools my customers now ask before they ask Google. Some of them are pages I am now building on purpose, because the AI already believes they exist and is sending people to look for them.

AI is now a traffic source that fabricates your sitemap. Treat its 404s as two things at once: a bug surface to clean up, and a wishlist someone else wrote for you. I already build this product mostly with AI agents (I even run them in the cloud), and now AI shapes how people find it too. The handful of lines it took to start listening have paid for themselves twice.