What is robots.txt?
TL;DR
robots.txt is a text file placed in a website's root directory that instructs search engine and AI crawlers which pages they are allowed or disallowed to crawl.
Definition
robots.txt is a text file placed in a website's root directory that instructs search engine and AI crawlers which pages they are allowed or disallowed to crawl. It is the first file crawlers look for when visiting a site.
Why It Matters
Properly configured robots.txt ensures search engines and AI systems can access your important content while blocking low-value or private pages. Misconfigured robots.txt can accidentally block crawlers from your entire site.
Examples
- Allowing all crawlers: User-agent: * / Disallow:
- Blocking a specific directory: User-agent: * / Disallow: /admin/
Frequently Asked Questions
Should I block AI crawlers in robots.txt?
Blocking AI crawlers (like GPTBot or CCBot) prevents AI systems from accessing your website for training and retrieval. For most businesses, allowing AI crawlers is beneficial - it improves your chances of being recommended by AI assistants. Only block if you have specific reasons (e.g., protecting proprietary content).
Related Terms
