Re: Need to block Web Spider software like Teleport pro

From: Rijesh (Rijesh_at_discussions.microsoft.com)
Date: 05/09/05 

Date: Sun, 8 May 2005 22:11:01 -0700

Currently the Teleport Pro have user agent label "Teleport Pro/1.29.1718”, so
how do I block access on the server level, if user agent is above? I know it
is not feasible, but it would be an approach towards blocking of such browser
access. Apache have directive to control access based on user agent, but I
don't find anything in IIS. Some times these type requirement are very much
needed.

Thanks,
Rijesh.

"Tom Kaminski [MVP]" wrote:

> There's a user agent string that gets sent by a web browser that identifies
> the browser type and version as well as client OS. Teleport pro can be
> configured to send any user agent info you desire so it appears to be just
> another browser.
>
> "Rijesh" <Rijesh@discussions.microsoft.com> wrote in message
> news:62CFCECA-99E4-46C6-97AA-C01EDB0054D9@microsoft.com...
> > What is Configurable Agent Identity?
> >
> > "Tom Kaminski [MVP]" wrote:
> >
> >> "Rijesh" <Rijesh@discussions.microsoft.com> wrote in message
> >> news:DA560DB6-C7B5-4F09-9927-D1076DD0816A@microsoft.com...
> >> > Hi,
> >> >
> >> > Recently our website crashed due to malicious activities by a stranger
> >> > by
> >> > using web spider software Teleport pro. Teleport Pro is the software
> >> > that
> >> > does this web capture activity, the client that was repeatedly hitting
> >> > us
> >> > every few seconds with the 'web capture' and ultimately caused for the
> >> > web
> >> > server crashes.
> >> >
> >> > I did some research on how to block web spider software like Teleport
> >> > pro.
> >> I
> >> > couldn't find any effective way to block it as it works like a browser
> >> > and
> >> > does the trick with HTTP functions. One way to block such activities is
> >> > by
> >> > blocking IP address, the way which we did. Yes, if they start it from
> >> another
> >> > IP address again we will be in trouble. So blocking IP address is just
> >> > a
> >> > temporary solution.
> >> >
> >> > I would like to know an effective way to prevent such software from
> >> > spidering my site and making the content available off line. Can anyone
> >> > provide some ideas to block such software activities? Is there any
> >> > patch,
> >> > method to block it with in IIS?
> >> >
> >> > There are some software available to protect the content that capable
> >> > to
> >> > block Teleport pro , The one is TagsLock,
> >> > http://www.tagslock.com/tagslock_pro.htm . But I am not sure how
> >> > efficient
> >> > this software and how it works. Did anyone tested it and give me the
> >> opinion?
> >> > Is that cause to decrease the performance?
> >> >
> >> > Ultimately I want a solution to block similar software activities in an
> >> > effective way and any advice from anyone would be grateful.
> >>
> >> Because it has this feature:
> >>
> >> "Configurable Agent Identity allows Teleport Pro to impersonate popular
> >> browsers; gets data from even the stingiest servers"
> >>
> >> it doesn't appear that you can do anything about it, other than monitor
> >> and
> >> block IP addresses. Why is your server crashing though? That shouldn't
> >> happen ...
> >>
> >> --
> >> Tom Kaminski IIS MVP
> >> http://www.microsoft.com/windowsserver2003/community/centers/iis/
> >> http://mvp.support.microsoft.com/
> >> http://www.iistoolshed.com/ - tools, scripts, and utilities for running
> >> IIS
> >>
> >>
> >>
>
>
>

Relevant Pages

  • Re: Need to block Web Spider software like Teleport pro
    ... There's a user agent string that gets sent by a web browser that identifies ... > What is Configurable Agent Identity? ... >> Tom Kaminski IIS MVP ...
    (microsoft.public.inetserver.iis.security)
  • Re: clj in course material
    ... part of this is on general purpose libraries. ... Slide img2.html ... As this related to UA string based browser sniffing why doesn't the ... a user agent send the same sequence of characters in its User Agent ...
    (comp.lang.javascript)
  • Re: jQuerys latest stab at competence
    ... User Agent header, and the specification for the HTTP 1.1 User Agent ... The UA string exposed to scripts is the same ... with the intention of fooling browser sniffing that had not ...
    (comp.lang.javascript)
  • Yet another appeal for browser sniffing
    ... are redirecting via server side User Agent sniffing. ... if the browser should be redirected or not into a hopefully ... Redirecting by server side UA sniff? ... "Features Detections Good Because ...
    (comp.lang.javascript)
  • Re: User agent fingerprinting
    ... submissions from forms), I was planning on logging a bit of data about ... Allthough I do not see the benefit of storing them as a md5. ... whether or not the browser is unique, I don't need to know the IP ... address, user agent, or other data that might identify a user. ...
    (comp.lang.php)