Robots.txt & Sitemap Validator
Verify search engine crawler directives, parse XML sitemaps, and diagnose indexing configuration errors.
Robots.txt & Sitemap FAQ & Guide
What is a robots.txt file?
A robots.txt file is a standard text file placed in the root directory of a website. It instructs search engine web crawlers (like Googlebot or Bingbot) which pages or folders they should or should not request from your site.
What is an XML Sitemap?
An XML sitemap is a structured file containing a list of URLs on your website. It allows search engines to find, crawl, and index all your public content efficiently, and provides helpful metadata such as when a page was last modified.
What does "Disallow: /" mean in Robots.txt?
The `Disallow: /` instruction forbids crawlers from fetching any page or content on the domain. This is often used on staging environments, but accidentally pushing it to production will cause search engines to de-index your entire site.
How should robots.txt declare sitemaps?
A sitemap can be declared anywhere in the robots.txt file by adding a line with the syntax `Sitemap: https://yourdomain.com/sitemap.xml`. This tells crawlers exactly where to find your map file without depending on default search pathways.