AI crawlers miss JavaScript-injected structured data

Your structured data can be valid, visible in Chrome and absent from the raw HTML an AI crawler receives.

That is the technical GEO problem in one sentence. The page works for a person with a browser. It does not necessarily work for the system that supplies an answer engine with evidence.

The page you see is not always the page the crawler sees

Most teams validate structured data in Google's Rich Results Test, see a passing result and close the ticket. That test answers a different question. Googlebot can render JavaScript. Many AI crawlers fetch the initial HTML and stop.

GPTBot, PerplexityBot, ClaudeBot and other crawlers may receive the document before hydration. They do not wait for a React component to mount, a tag manager to inject JSON-LD or a third-party widget to load an FAQ. If the useful context arrives after the first response, the crawler may never receive it.

This creates an awkward mismatch. The page can look complete to a buyer and still provide an incomplete evidence set to an answer engine.

Start with the raw response

Run the same template twice: once in a browser with JavaScript enabled, and once as raw HTML with JavaScript disabled. Compare the word count, headings, canonical tag, product attributes and JSON-LD. The difference is the part an AI crawler may miss.

Make that comparison part of deployment rather than an annual audit. A new client-side component can remove the information a crawler needs without changing what the browser appears to show.

Access comes before markup

The first check is not schema. It is whether the crawler can reach the page.

Robots.txt and server rules

Enterprise sites often block unfamiliar bots by default. That can include the crawlers that supply the answer engines your buyers use. Check robots.txt, server headers, IP allowlists and edge security rules together. A robots file that permits a crawler is not useful if the firewall rejects the request.

Keep the policy deliberate. Decide which crawlers the business wants to serve, document the decision and test the response from outside the browser.

Server-side or static output

Structured data needs to exist in the initial HTML. Server-side rendering, static generation and carefully implemented middleware can provide it. A client-only application cannot assume that every retrieval system will render the page for it.

Schema with a job to do

Once the page is reachable and readable, give the crawler useful context. A commercial page may need Product or Service schema, Organization identity, BreadcrumbList and, where appropriate, Article, FAQPage or HowTo markup. The markup should describe what the page says. It should not be an inventory of every schema type available.

llms.txt is a guide, not a substitute

A plain-text llms.txt file can point AI systems towards the pages and concepts a domain considers important. It is not a formal standard, and adoption is voluntary. That makes it useful as a clear editorial map, not as a replacement for crawlable HTML, robots policy or schema.

# Inflexion AI Discovery Guide
# https://inflexionmedia.com/llms.txt

## Core pages
- https://inflexionmedia.com/ : Homepage
- https://inflexionmedia.com/aeo/ : Organic and AI Search
- https://inflexionmedia.com/technical-geo/ : Technical GEO
- https://inflexionmedia.com/ai-visibility-analytics/ : Search Intelligence

The file is easy to create. Keeping it accurate is the work. A link to a page that has moved, a service description that no longer matches the page or a list that omits the current priority weakens the guide.

The audit should end in an order of work

A technical GEO audit is useful when it changes what a team does next. Our sequence is:

1. Request the raw HTML and rendered page for each important template.
2. Check crawler access across robots.txt, headers and edge controls.
3. Compare the browser and raw HTML for content, links, canonical data and schema.
4. Check schema coverage against the page's actual commercial purpose.
5. Review llms.txt and the XML sitemap for accuracy.
6. Use crawler logs to see which bots visited, what they fetched and where they stopped.

The usual priority is access first, rendering second and schema third. The order matters. Perfect markup on an inaccessible page is still missing evidence.