Crawler readiness
AI crawler readiness checklist
A pre-launch checklist for crawler access, answer-engine evidence, EEAT proof, FAQ depth, service/location coverage, and useful llms.txt context.
Crawler readiness
Separate access from authority
- Confirm the public homepage returns a successful fetch for the URL you intend to analyze.
- Review title, description, and OpenGraph-style metadata for the primary landing page.
- Check robots.txt for intentional allow/deny policy and a declared sitemap.
- Confirm /sitemap.xml or the declared sitemap exposes crawlable paths.
- Look for FAQPage, Person, Organization, Service, LocalBusiness, and review-style proof where those entities fit the site.
- Note whether llms.txt is curated context or just a sitemap-like URL list.
Crawler readiness
Do not overclaim
- Treat the technical crawl score as access/indexability hygiene, not AEO strength.
- Only scan public sites you own, control, or have permission to inspect.
- Read top_gaps before chasing the top-level score.
- Treat llms.txt as an emerging signal, not a universal crawler standard.
Crawler readiness
Named outputs to inspect
- score and technical_crawl_score show whether public resources are reachable and parseable.
- aeo_authority_score shows answer-engine authority, EEAT, FAQ depth, topical coverage, and curated llms.txt quality.
- top_gaps[] explains the highest-priority issues found in the public-page analysis.
- Use RapidAPI for marketplace auth, plans, and playground testing after the checklist.