Robots Exclusion
Robots Exclusion provides step-by-step instructions with regard to how a site may be indexed by an indexing robot, or that a resource discovery robot has visited part of a site that for some reason shouldn't be visited by robots.
Visit robots.txt


