Acme.NoRobots

Acme.NoRobots is a Robot Exclusion Standard that allows each web server to set up a single "/robots.txt" file with pathnames that robots should not look at.

More information about them, from the website:

The basic idea of the Robot Exclusion Standard is that each web server can set up a single file called "/robots.txt" which contains pathnames that robots should not look at. See the full spec for details. Using this class is very simple - you create the object using your robot's name, and then you call ok() on each URL. For efficiency, the class caches entries for servers you've visited recently.

Visit Class Acme.NoRobots