How to audit website metadata for AI search and crawler readiness

Start with signals a crawler can actually see

Before chasing AI search visibility, verify that the public page fetches, describes itself consistently, exposes crawl policy, and points to the pages worth discovering.

• Fetch the canonical public URL and record the HTTP status.
• Inspect title, meta description, canonical URL, and OpenGraph fields.
• Check robots.txt for intentional allow and deny rules.
• Verify that a sitemap exists and exposes crawlable URLs.

Machine-readable context reduces ambiguity

Structured data and llms.txt are not magic ranking switches, but they help downstream systems interpret what the page represents and which content is intentionally available.

• Add JSON-LD for the entity type that best matches the page.
• Keep sitemap entries aligned with canonical URLs.
• Use llms.txt to summarize public documentation and tool routes.
• Treat readiness scores as diagnostics, not ranking promises.

Re-run after one patch

The useful metric is not the first score. It is whether a visible metadata, sitemap, structured-data, or llms.txt patch moves the evidence-backed gap list in the right direction.