Consumer-to-Business (Crowdsourced) Data Security Project
I'd like to create & add a web plugin functionality allowing to identify "bad bots", which disobey ROBOTS.TXT protocol. Identification of such "bad bot" on one site will allow to distribute bot's ID/IP to anyone on the network. When the same bot will show up on your server - it will be auto banned
Q1: It would be also interesting to create functionality which will be automatically banning bad bots ...
Idea Description
All data collected to from multiple sites will be provided to all private users free of charge. It will be also made available to corporations through paid subscription.
IP/ID of "bad" bots, which disobey robots.txt, will be aggregated at one central database at http://botboot.com and then will be made available to all members, who will be able to decided if they want to ban all "bad bots" through HTACCESS
The plugin's functionality will also protect site's content from illegal screen scraping.
What will you do if you win $10,000 for this idea?
1. Make a deposit to my programmers so they could develop functionality of detecting bad bots and pass and store all information to botboot.com
2. connect to all leading open source projects and provide them code to implement bot traps into their next release.
3. Hire good copywriter to write site's content and prepare a proposal to corporate clients.
Vote for it now.
Search bots spy for information... there are certain rules where they can/cannot look..... "good" bot follow those rules, "bad" bots do not... Since bots travel from server to server this means, that it will be enough to identify it once, then pass this info to other websites ... when this "bad " bot arrives to protected site, it will be bounced.