Robot Exclusion

Definitions

1. The practice of using mechanisms like `robots.txt` to signal which parts of a site automated crawlers should avoid indexing or scanning. Robot exclusion is a courtesy convention, not a security boundary, so sensitive paths listed there may actually attract attention from attackers.

   In plain English: A way of asking web crawlers to avoid certain paths, but not a real security control.

   Example: "The admin backup directory should never have been treated as protected just because robot exclusion told search engines not to index it."

Related Terms

• Credential Leak
• Data Leakage
• Health Check Security
• Internet Exposure
• Known Vulnerability
• Port Forwarding
• Public Vulnerability
• Rogue Server