Skip to content
weballin

weballin DIGIWIKI

Robots.txt

robots.txt is a text file at a site’s root that tells supporting crawlers which paths they may crawl. Blocking crawling does not guarantee removal from search results, and an accidentally deployed Disallow rule can block an entire site.

weballin

robots.txt controls crawler access. It can help limit crawling of paths such as internal search results and large sets of parameter URLs. It is not an access-control system for confidential information.

Blocking a known URL does not necessarily remove it from search results: a search engine may still discover and list it through other links. To apply noindex, the crawler must be able to access the page and read that directive.

Check robots.txt before deployment. Leaving the development rule `Disallow: /` in production can block crawling across the site. Blocking CSS or JavaScript needed to render a page can also prevent search engines from understanding its content.

Key takeaways

  • It specifies which paths supporting crawlers may access.
  • It does not guarantee removal from search results.
  • It is served from `/robots.txt` at the site root.
  • Keep important pages and rendering resources accessible.

References

Frequently asked questions

Why can a blocked page still appear in search results?

robots.txt controls crawling, not removal from the index. A search engine may discover a blocked URL through other links. If you use noindex, allow the crawler to access the page so it can read the directive.

Should CSS and JavaScript be blocked in robots.txt?

Keep the resources needed to render the page accessible. Blocking them can prevent Googlebot from understanding the page correctly. Limit crawling selectively rather than blocking essential resources.

Related terms