Back to Top

Wednesday, November 29, 2006

How to create a well behaved bot?

From the ever watchful master of bots and scrapers incrediBILL (:-)) comes the following tip, which you might find useful if you yourself plan / need to implement a web crawler as part of your product / service.


Post a Comment

You can use some HTML tags, such as <b>, <i>, <a>. Comments are moderated, so there will be a delay until the comment appears. However if you comment, I follow.