← Back to Baljia AI

Free tool. No sign-up. No email.

Can Google and AI crawlers read your site?

Enter your website for a quick pass/fail on the basics: robots.txt, AI crawler rules, your sitemap, the homepage title and meta description, and llms.txt. Results show right here.

What it checks

  1. robots.txt exists. The plain-text file crawlers read first. It also tells them where your sitemap is.
  2. AI crawlers are not blocked. Whether your robots.txt keeps AI crawlers off your homepage: the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), the fetchers that visit when a user asks (ChatGPT-User, Perplexity-User), and the training tokens (GPTBot, ClaudeBot, Google-Extended). Google-Extended only controls Gemini training and grounding, not Google Search. OpenAI and Perplexity say their user-triggered fetchers may not follow robots.txt.
  3. A sitemap exists. At /sitemap.xml, or wherever robots.txt says it is. It lists the pages you want found.
  4. The homepage has a title and a meta description. Read from the HTML your server sends, which is what most crawlers see.
  5. llms.txt (optional). An emerging convention: a short Markdown summary of your site for AI tools. It is not a standard, and Google has said it does not use it, so a missing file is not a failure.

What it does not tell you

  • robots.txt is only your stated policy. A firewall or CDN setting can still turn an AI crawler away, and this check does not test that. If you are unsure, ask your host or CDN whether AI crawlers are allowed.
  • Passing the basics does not mean you will rank or be cited. They are the floor, not the result.
  • It reads your homepage only, plus robots.txt, the sitemap and llms.txt. No AI is involved.
  • We fetch these files once, as "BaljiaSiteCheck/1.0", and do not store the results.
  • To keep it free, each visitor can run 10 checks an hour, with a daily limit for everyone.

Go further