Robots.txt is a file on a website that tells search engines which areas they may crawl and which they may not. It usually sits in the root directory of the domain, for example at yourdomain.com/robots.txt.
The file is aimed primarily at search engine bots such as Googlebot.
How does robots.txt work?
In robots.txt you can block particular pages, directories or file types from search engines.
Typical examples:
- login pages
- internal search results
- test pages
- filter or parameter pages
This stops search engines from crawling content that is unnecessary or unimportant.
One thing to be clear about: robots.txt does not automatically prevent indexing. If a blocked URL is linked from other pages, it can still appear in the results.
Why robots.txt matters
It helps you use your crawl budget sensibly and point search engines at the content that matters.
On large websites, online shops or sites with many filters and parameters in particular, it can keep unnecessary pages out of crawling.
Mistakes in robots.txt can also do the opposite and accidentally block pages you need indexed.