Reply by agallant

  • Advertise with us

Posted on Our Lumberjocks forum threads are appearing in other sites!! Oh No! Not again!

View agallant's profile


551 posts in 3942 days

#1 posted 03-01-2011 04:05 PM

I was the IT manager at a web site that was getting about 18M hits a month and we had to deal with this issue all of the time. The issue with scraping is that it puts a huge load on your servers. Most companies limit the amount of hits that can come from one source. Other things we did was block IPs from outside our market. We had all of Asia and Africa blocked.

Scraping is used to gather data do post on another web site. From there what they do with the data is endless. Some people use the data to run their own web site allowing the site they scraped to do the heavy lifting of gathering the actual data other people use it to get hits to deploy viruses, the list goes on and on of what they use the data for.

DISCLAIMER: Any posts on LJ are posted by individuals acting in their own right and do not necessarily reflect the views of LJ. LJ will not be held liable for the actions of any user.

Latest Projects | Latest Blog Entries | Latest Forum Topics