Robots.Txt Glossary

Browse 1 robots.txt terms defined in plain English, from the cultural dictionary of computing.

1 Robots.Txt Terms

Robot Exclusion

The practice of using mechanisms like `robots.txt` to signal which parts of a site automated crawlers should avoid indexing or scanning. Robot exclusion is a...

Related Topics

• Exposure (1 terms in common)
• Web Crawlers (1 terms in common)