Crawling is the process search engines use to explore the web. Programs known as crawlers or bots request pages, read their content and follow the links on them to the next page. The best-known crawler is Googlebot.
Crawling is the first step before a page can appear in search results at all. Anything that is never crawled cannot be found.
How does crawling work?
The crawler starts with known addresses and discovers new pages through links and the XML sitemap. The robots.txt file sets out which areas it may visit. Problems arise mainly from:
- pages without internal links that the crawler never finds
- broken links and long redirect chains
- slow servers that hold back requests
- content that only loads via JavaScript and is therefore hard to read
On large websites such as online shops with thousands of filter pages, crawl budget becomes important: Google does not crawl without limits but spreads its requests.
Why does crawling matter?
Google is not the only one crawling. AI providers such as OpenAI and Perplexity send out their own crawlers to collect content for their answers. If you block them in your robots.txt, you are less likely to appear as a source in those answers.
With ongoing SEO management, you regularly check whether Google and AI crawlers reach your most important pages – before missing visibility costs you revenue.
