From: Cameron Kaiser (spectre_at_floodgap.com)
Date: 2004-02-06 17:16:13
> > > > - Is there an easy way to download the whole archive? > > > > > > If you have curl, and don't need the attachments... otherwise use your > > > favourite browser's archive making facilities. > > > > May I say that as someone who pays for my bandwidth, I hate people who do > > this, and I will block them indefinitely from my site if I see activity > > of this sort. This drives up load average, diminishes outgoing traffic, and > > is just plain rude. > > err isnt the purpose of websites that they are beeing downloaded? especially > if they are archived? but maybe thats just me ... :=P Don't be deliberately obtuse. I'm referring to people with abominations like Offline Explorer or WebZIP that can, when parallelized, bring servers to their knees. If someone wants to view the site, fine. Rapidly downloading it to disk prevents others from accessing the site during that period, and on the size that services like the HyTelnet gateway or Commodore Knowledge Base and Secret Weapons are, locks my 384Kbps uplink for unacceptably long periods. Having had this happen multiple times, if I notice a particular host exhibiting spider-like activity that I don't allow (well-behaved bots likeo Googlebot are specifically excepted), I ban IP blocks first and ask questions later. Incidentally, to robots.txt suggesters, the whole selling point of many of those tools is that they ignore robots.txt. Good bots obey <meta> tags and stay out of where they're not wanted. It's the badly behaved bots that I'm banning. Bluntly, I'm not buying extra bandwidth for degenerates who believe that having a local copy of the entire Internet is their right over all other regular requests, including from whom I bought the line for in the first place -- me. -- ---------------------------------- personal: http://www.armory.com/~spectre/ -- Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * firstname.lastname@example.org -- The things which hurt, instruct. -- Benjamin Franklin ---------------------- Message was sent through the cbm-hackers mailing list
Archive generated by hypermail pre-2.1.8.