Audits and the crawler

An audit crawls the site, saves each page's HTML, runs the checks, scores the result, reconciles issues with the previous audit, and then runs the AI analysis. The deterministic part always completes on its own: if the AI step is skipped or fails, you still get every finding, issue and the score.

Each audit has an Activity log listing every page fetched, check run, AI step and tool call, with timings and cost.

Set these per site in the site's Settings tab.

• Links from the start URL, breadth-first, up to the depth and page limits.
• Sitemaps from robots.txt Sitemap: lines and /sitemap.xml, following up to 8 child sitemaps and 5,000 URLs.
• Pages disallowed by robots.txt are skipped, and requests to a host are at least 0.5 s apart, or the robots.txt Crawl-delay if that is longer.

With rendering on auto, a page is re-rendered in a headless browser only when its HTML looks like an empty app shell: very little visible text plus single-page-app or site-builder markers. Server-rendered pages are checked as delivered. Use force to render every page (within the render budget) or off to never render.

When a firewall answers with a block (403, 429, 503 and similar, recognised for Cloudflare, Akamai, Imperva, Sucuri, AWS WAF and others), the crawler waits and retries, then slows down for that host. Pages that stay blocked are marked as blocked and reported once as crawl.blocked. They are not counted as errors and don't affect the score.

• CSV: issues or recommendations, for spreadsheets.
• JSON: the whole audit (score, issues, resolved issues, recommendations, diagnoses), for scripts and AI agents.

Both are on the audit page. Agents can also read everything through the [MCP server](/docs/mcp).