Site Admin Stuff

Stuff I depend on as a webmaster and web developer

Go home

Robots / Spiders / Search

Robots / Spiders / Search

Handling robots or adding search to your site
Robots.txt Robots.txt Keep robots out with the Robots.txt file
Who's knocking on the door? Who's knocking on the door? Explaination of robots.txt & the nofollow meta tag, has examples
Database of robots
Web robots FAQ
Robots.txt generator Robots.txt generator From SEOChat
htdig Free crawler
Lucene/Nutch Lucene/Nutch Apache's crawler software (free) {Java}
Robot Trap How to keep bad robots, spiders and web crawlers away
Php Dig Php Dig Web spider & search engine [php & mysql]

Please report broken links and resources that you think bite.

Own only what you can carry with you; know language, know countries, know people. Let your memory be your travel bag. -Aleksandr Solzhenitsyn
This page might be ugly, but it's valid HTML 4.0 Strict and CSS. Yay!
© Copyright 2005-6. A David Production.

You may not copy, transmit, redistribute, publish, modify, create derivative works or exploit in any way any of the content of this website without permission, except where expressly permitted by copyright law. Thank you.