Are You Accidentally Blocking Ask.com on Old Websites?

In my mailbox today

Re: [url]

To whom it may concern:
We’d appreciate your forwarding this to the appropriate contact at [url]

….. we have noticed that you have blocked the Ask.com agent from crawling your site in the robots.txt file.

We definitely understand that granting our crawler access to your site is an act of trust, which we will use with utmost integrity. So we are asking you to reconsider allowing our crawler access to your site. If there is some reason you are blocking us, please share that with us and allow us to address it. Or if there is no reason, then we ask you to go ahead and remove the line in your robots.txt file that is blocking us. Our goal is to have the same access you have granted other search engines.

The line in question is:

User-agent: Jeeves
Disallow: /

Nice!

BUT…. long ago, Jeeves was used by Leon Brocard to create a web mirroring bot.

Therefore “Jeeves” used to be listed as

robot-id: jeeves
robot-name: Jeeves
robot-cover-url: http://www-students.doc.ic.ac.uk/~lglb/Jeeves/
robot-details-url:
robot-owner-name: Leon Brocard
robot-owner-url: http://www-students.doc.ic.ac.uk/~lglb/
robot-owner-email: lglb@doc.ic.ac.uk
robot-status: development
robot-purpose: indexing maintenance statistics
robot-type: standalone
robot-platform: UNIX
robot-availability: none
robot-exclusion: no
robot-exclusion-useragent: jeeves
robot-noindex: no
robot-host: *.doc.ic.ac.uk
robot-from: yes
robot-useragent: Jeeves v0.05alpha (PERL, LWP, lglb@doc.ic.ac.uk)
robot-language: perl5
robot-description: Jeeves is basically a web-mirroring robot built as a
final-year degree project. It will have many nice features and is
already web-friendly. Still in development.
robot-history: Still short (0.05alpha)
robot-environment: research

modified-by: Leon Brocard

I don’t know why the bot was on the “evil list” but it explains why this old site had it blocked.

I had always had my eye open for

Ask Jeeves
Ask Jeeves/Teoma
DirectHit

…but never just “Jeeves” - but it is definitely an alias.

I thought it was awesome for Ask.com to take the time to crawl these files and send out notes.

del.icio.us Digg it ma.gnolia Netscape reddit StumbleUpon Yahoo MyWeb

Leave a Reply