Robots Exclusion

Tagged:  

Robots Exclusion provides step-by-step instructions with regard to how a site may be indexed by an indexing robot, or that a resource discovery robot has visited part of a site that for some reason shouldn't be visited by robots.

Visit robots.txt

Share/Save