Polybot

Tagged:  

Polybot is a high-performance distributed web crawler developed in the Department of Computer Science and Engineering at Polytechnic Institute of New York University as part of an academic research project that explores new techniques for searching and analyzing the World Wide Web.

Visit NYU-Poly

Share/Save