Robots meta tags are a precise page-level crawling and indexing control tool. Unlike robots.txt, which applies to the entire path, meta tags can give different instructions for individual pages. Key directives: noindex (exclude index), nofollow (do not follow links), nosnippet (block snippets), noimageindex (exclude image index), max-snippet (-1 means no length limit).
The most common mistake is applying noindex to pages that have been blocked from crawling with robots.txt. If Googlebot cannot crawl, the meta tag itself cannot be read, so the noindex instruction is not processed. This can result in a situation where only the page URL remains in the index (without any content).
In the weballin standard, noindex is applied by default to pages with low index value, such as temporary campaign pages, internal search results, and filter URLs, and the meta robots status of major page templates is verified using a checklist after deployment or a redesign. In the SPA/Next.js environment, use a URL inspection tool to check whether noindex is correctly included in the server-rendered HTML.
Key takeaways
- Control index, snippet, and link following on a page-by-page basis.
- Directives such as noindex, nofollow, nosnippet, and noimageindex are supported.
- When robots.txt is blocked, crawling itself is blocked and meta tags cannot be read.
- To target a specific crawler, use its supported name (e.g. `name="googlebot"`).
- If noindex is accidentally left on important pages after deployment, search exposure will disappear.