Crawl controls do not provide privacy
Decide whether the goal is controlling crawling or excluding a page from indexing before choosing a directive. Blocking crawling can prevent a crawler from reading a page-level instruction.
Robots.txt communicates crawl instructions to cooperating crawlers. A noindex instruction concerns search inclusion when it can be processed. Neither should be used as the sole protection for confidential content. Authentication and correct authorization address a different problem.
A worked example
A hypothetical staging directory is disallowed in robots.txt but remains publicly accessible to anyone with its URL. That is not private staging. On another page, a crawler block prevents retrieval of the noindex directive the owner expects the search system to see.
How to apply it
- State whether the objective is crawl management, index exclusion or actual access protection.
- Use the appropriate control and test the response from the intended access context.
- Check for conflicting rules before assuming a search platform has processed the instruction.
What the result can establish
For public pages, verify intended crawl and indexing behavior separately. For private areas, verify that unauthorized requests cannot obtain the content. Do not publish credentials or client data on the assumption that a crawler directive makes the URL secret.
Reference: Google: robots.txt.
Supporting documentation
Google: robots.txtGoogle: noindexGoogle: technical requirements