Skip to content
weballin

weballin DIGIWIKI

X-Robots-Tag

X-Robots-Tag supplies indexing and search-display instructions in an HTTP response header. It works for files such as PDFs and images, but does not control access or guarantee immediate removal.

weballin

A public PDF can return X-Robots-Tag: noindex to request exclusion from search. That does not remove direct links or download access. Private material needs separate controls such as authentication.

Google must be able to recrawl the file to read the header. Blocking it in robots.txt can prevent that, and processing changes in search takes time.

An Apache example is Header set X-Robots-Tag "noindex", without a colon after the header name. Nginx uses directives such as add_header X-Robots-Tag "noindex"; in an appropriate server or location configuration and does not read .htaccess. Limit the scope and inspect actual responses.

References

Frequently asked questions

How can I check which resources are affected?

Inspect response headers for both target and exempt files using developer tools or an HTTP client. Check CDN caching and duplicate headers; the presence of a configuration entry alone does not prove application.

Related terms