http://www.chemspider.com/open-chemistry-web/chemrefer-and-robotstxt.html The Project: Open and Integrated Structure and Text Search Sun, 19 Apr 2009 16:13:14 +0100 http://wordpress.org/?v=2.8.4 hourly 1 http://www.chemspider.com/open-chemistry-web/chemrefer-and-robotstxt.html/comment-page-1#comment-129 ChemSpider Blog » Blog Archive » Support from the Publishing Community for Text Indexing Scientific Articles Thu, 06 Dec 2007 02:05:31 +0000 http://www.chemspider.com/open-chemistry-web/?p=3#comment-129 [...] few weeks ago Will Griffiths blogged about robots.txt files for informing on indexing policies for a website and then later on our discussions with RSC. As a result of our experiences in the [...] [...] few weeks ago Will Griffiths blogged about robots.txt files for informing on indexing policies for a website and then later on our discussions with RSC. As a result of our experiences in the [...]

]]> http://www.chemspider.com/open-chemistry-web/chemrefer-and-robotstxt.html/comment-page-1#comment-8 David Bradley Tue, 16 Oct 2007 11:22:54 +0000 http://www.chemspider.com/open-chemistry-web/?p=3#comment-8 As I understand it, robots.txt is taken by search engines to be nothing more than a set of suggestions. Several people in the SEO community report that SEs sometimes ignore robots.txt altogether and some spiders don't even have the ability to read it in the first place let alone respond to it. robots.txt is certainly not a legal document. If you post information on the web you must accept that it will be spidered. The only full-on way to stop a spider in its tracks is to ban its IP address from your server. But, that opens a minefield as some spiders have multiple IPs and some are dynamic, so what could be the IP address of a spider you wish to ban one day could be a potential visitor the next. As I understand it, robots.txt is taken by search engines to be nothing more than a set of suggestions. Several people in the SEO community report that SEs sometimes ignore robots.txt altogether and some spiders don’t even have the ability to read it in the first place let alone respond to it. robots.txt is certainly not a legal document. If you post information on the web you must accept that it will be spidered. The only full-on way to stop a spider in its tracks is to ban its IP address from your server. But, that opens a minefield as some spiders have multiple IPs and some are dynamic, so what could be the IP address of a spider you wish to ban one day could be a potential visitor the next.

]]>