SEO Agent
Sign in

AISEOBot

AISEOBot is the crawler behind this SEO audit service. It visits a website only when the site's owner (or someone working for them) adds it and starts an audit, and checks pages the way a search engine would: titles, links, structured data, speed and similar.

How it identifies itself

User-Agent: AISEOBot/1.0 (+https://ai-seo.vdev.web.id/bot)

How it behaves

  • It follows robots.txt (rules for AISEOBot or *) and its Crawl-delay.
  • It fetches a limited number of pages per audit with a few parallel requests, spaced out per host.
  • If a firewall or rate limit pushes back (403/405/429 with a block page), it slows down and retries a couple of times, then stops rather than keep asking.
  • It only sends GET requests and never submits forms or logs in.

Allowing or blocking it

To let audits through a web application firewall (Cloudflare, Alibaba Cloud WAF, Akamai and others), add an allow rule for requests whose user agent contains AISEOBot. To keep it out, add this to robots.txt: 

User-agent: AISEOBot
Disallow: /