Are spiders or robots Legal?

[bluewaves]

Is the use of automated software’s which crawl through sites, and are used to collect information from online data bases legal?
I have heard that some sites have a robot.txt file which redirects or forbids the robot from scavenging certain information? Is the robot.txt file on firm ground legal
to stop actions of these types of software’s?

 

[Pfaffing85690]

spiders and robots are scripts, not applications or software. And yes, they are legal. A robot.txt or spider.text file simply stops this download capability.

Most sites do not use such files anylonger because by their use they exclude being listing on search engines such as google, Mr. Jeaves and Yahoo.

Quite counter productive huh?

And a simple way to stop such crawling would be to use a web language such as Cold Fusion, php, asp or other dynamic lanugage which does not produce the page until called.

Oh, and database is one word.

 

[mykoleary]

spiders and robots are scripts, not applications or software. And yes, they are legal. A robot.txt or spider.text file simply stops this download capability.

Point of reference: these files are SUPPOSED to allow site owners control, but there is nothing that can stop the many badly behaving robots that ignore the files...

 

[Pfaffing85690]

Want to bet

 

[bluewaves]

Thank you for the replies.

I have another question. Is the sale of a spider, which goes through a proprietary publicly viewable online database illegal in USA law? For example, XYZ corporation sells a spiders which extracts and organize information from the yellow pages directory. The spider in this example extracts data on XYZ client’s computer. The software from XYZ corporation is used by marketers to solicit business.
Is the sale of the spider script illegal for XYZ corporation on the basis that it is like selling the Proprietary database itself. Secondly, is the use by marketer to collect information from such a database illegal on the basis that they are using a Proprietary yellow pages database?

---Thank you.

 

[mykoleary]

Re: Thank you for the replies.

I have another question. Is the sale of a spider, which goes through a proprietary publicly viewable online database illegal in USA law? For example, XYZ corporation sells a spiders which extracts and organize information from the yellow pages directory. The spider in this example extracts data on XYZ client’s computer. The software from XYZ corporation is used by marketers to solicit business.
Is the sale of the spider script illegal for XYZ corporation on the basis that it is like selling the Proprietary database itself. Secondly, is the use by marketer to collect information from such a database illegal on the basis that they are using a Proprietary yellow pages database?

---Thank you.

This is likely illegal. Check the TOU if these online DB sites, many of which expressly forbid the scenario you posit.