Re: Restrict by UserAgent

Exactly.

It's better to devote resources to harden the overall application and
architecture, than to go after a "moving target" like a User-Agent value (or
any header data as you have noted).

It's my first post in this community - and it's been a pleasure. Thanks!

-----------
Cheers,
Ed


"David Wang [Msft]" wrote:

> Unfortunately, when talking about "security" it tends to be more like the
> 99.999% vs 0.001% rule, where you are hunting for the couple of bad needles
> in the haystack.
>
> In general, the anonimity afforded by the Internet makes it very useful for
> malicious actions, especially when disguised in the midst of good ones.
>
> Thus, I would never base security decisions based on any request header
> value because that is unvalidated information from the client, the very
> source that you do not trust. This is why authentication is important when
> it comes to security because without it, you are merely talking about some
> imperfect heuristic to distinguish the good guys from the bad guys, and
> there will be false-positives. To reduce the false-positives, you will
> always end up a step behind the bad guys...
>
> --
> //David
> IIS
> http://blogs.msdn.com/David.Wang
> This posting is provided "AS IS" with no warranties, and confers no rights.
> //
>
> "Ed" <Ed@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
> news:A1C218F7-2EFF-4C35-B9A1-AEC43196FE3B@xxxxxxxxxxxxxxxx
> > Thanks again for your very helpful comments!
> >
> > What I meant by "performance concerns"/tradeoff is whether we *do* want to
> > start filtering because of these suspicious requests or just let it go
> > (don't
> > filter for UserAgent). It's somewhat like the 80/20 rule - do we want to
> > affect 80% of our users because of the actions of 20%? It's actually more
> > like 97% vs. 3%...We'll have to dig a little bit more on what the true
> > impact
> > this 3% is creating before doing anything....
> >
> > The other reality is that we should probably be looking somewhere else.
> > The
> > more I think about it, the less effective filtering by UserAgent becomes
> > as a
> > security option. I mean, if we're really dealing with some malicious
> > intentions, it's probably trivial to change the UserAgent info of a bot
> > anyway - I've already seen it discussed in some Java sites in an attempt
> > to
> > circumvent Google restrictions.
> >
> > -----------
> > Cheers,
> > Ed
> >
> >
> > "David Wang [Msft]" wrote:
> >
> >> Yeah, I was just thinking about the post I made in early morning and
> >> realized that URLScan isn't exactly a valid solution even if it is
> >> supported.
> >>
> >> Stitching together the two code samples I have will be sufficient, and it
> >> is
> >> specifically tuned to just this situation. There won't be any gotchas or
> >> performance issues because it is doing exactly what you need and nothing
> >> more.
> >>
> >> In other words, your performance concerns about filtering all requests
> >> for
> >> UserAgent is odd because... in order to restrict by UserAgent, don't you
> >> have to filter all requests for it *anyway*, so even if there is a perf
> >> concern doing this, don't you have to accept it if you want to filter?
> >>
> >> Performance concerns for the Apache solution is real because in that
> >> case,
> >> it is a general-purpose module being reconfigured for this task, which
> >> carries along with it a real performance caveat due to how it functions.
> >> However, I would not automatically assume that the same caveat affects
> >> IIS
> >> because it doesn't, especially with a specialized module and centralized
> >> configuration.
> >>
> >> --
> >> //David
> >> IIS
> >> http://blogs.msdn.com/David.Wang
> >> This posting is provided "AS IS" with no warranties, and confers no
> >> rights.
> >> //
> >>
> >> "Ed" <Ed@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
> >> news:ECF65CCC-9826-486B-8569-7C847B6423F7@xxxxxxxxxxxxxxxx
> >> > Hi again David,
> >> >
> >> > Well, after going through the available documentation on URLScan 2.5,
> >> > it
> >> > doesn't look like it'll serve this purpose at all. My interpretation of
> >> > the
> >> > documentation for the [DenyHeaders] settings is that it takes in a
> >> > header
> >> > NAME and not a header VALUE nor a header NAME/VALUE entry.
> >> >
> >> > In other words, it'll accept a setting like: "User-Agent:"
> >> >
> >> > but not these:
> >> >
> >> > "User-Agent: Java/1.5.0_02" ; name/value
> >> > "Java/1.5.0_02" ; value only
> >> >
> >> > Again this is only based on my understanding/perception of the
> >> > [DenyHeaders]
> >> > section of URLScan ini - couldn't really find any related
> >> > documentation/samples....
> >> >
> >> > Addtionally, the "User-Agent:" setting really seems like a disastrous
> >> > setting in my book since it translates to "deny all requests that
> >> > contain
> >> > a
> >> > User-Agent header" - equates to probably all known browsers. If this
> >> > interpretation is accurate, it's somewhat counter intuitive to IIS -
> >> > what's
> >> > IIS for otherwise*?
> >> >
> >> > *I do remember having to install IIS on a SQL server just to create a
> >> > certificate and have the option to secure SQL traffic (SSL) - may not
> >> > have
> >> > been the only way to do this, but after wasting a few hours trying to
> >> > get
> >> > this to work without using the IIS gui for this purpose, well, it
> >> > breaks
> >> > down
> >> > to "just do it"...
> >> >
> >> > -----------
> >> > Cheers,
> >> > Ed
> >> >
> >> >
> >> > "David Wang [Msft]" wrote:
> >> >
> >> >> Sure, you can do this by either:
> >> >> 1. Use an existing module to do this
> >> >> 2. Write some custom module to do this
> >> >>
> >> >> FYI: Apache cannot do this without using a custom module which happens
> >> >> to
> >> >> be
> >> >> widely distributed with it. Similarly, IIS can also cannot do this
> >> >> without
> >> >> using a custom module, but no one really provides a freely distributed
> >> >> one.
> >> >>
> >> >> Some that may work are from:
> >> >> iismods.com
> >> >> isapirewrite.com
> >> >>
> >> >> Personally, I would use URLScan since it is an existing, supported,
> >> >> and
> >> >> available module. It is really not different than how you do it on
> >> >> Apache.
> >> >> With Apache, you are simply configuring some pre-bundled module -- so
> >> >> how
> >> >> is
> >> >> that different than configuring a self-selected URLScan module on IIS,
> >> >> other
> >> >> than you having the freedom of choosing the URLScan module over any
> >> >> other?
> >> >>
> >> >> For those interested in custom code, you should be able to stitch
> >> >> together
> >> >> code between these two blog entries to do it.
> >> >> - Pick out requests based on a header (user-agent:)
> >> >> http://blogs.msdn.com/david.wang/archive/2005/08/03/HOWTO_ISAPI_Filter_logging_request_URL_and_headers_based_on_User_Agent.aspx
> >> >> - Deny requests based on a header (referer:)
> >> >> http://blogs.msdn.com/david.wang/archive/2005/07/01/HOWTO_ISAPI_Filter_rejecting_requests_from_SF_NOTIFY_PREPROC_HEADERS_based_on_HTTP_Referer.aspx
> >> >>
> >> >> --
> >> >> //David
> >> >> IIS
> >> >> http://blogs.msdn.com/David.Wang
> >> >> This posting is provided "AS IS" with no warranties, and confers no
> >> >> rights.
> >> >> //
> >> >>
> >> >> "Ed" <Ed@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message
> >> >> news:FF9735F2-E9D8-415B-A177-BF1637F9E6FA@xxxxxxxxxxxxxxxx
> >> >> > What was a "trivial occurrence" is now a cause for concern. We've
> >> >> > been
> >> >> > seeing
> >> >> > increased activity which seem to be from Java based
> >> >> > crawlers/spiders/scrapers.
> >> >> >
> >> >> > User agent is of the form: Java/[various versions]
> >> >> >
> >> >> > Is there a way to restrict/deny requests by a specific useragent on
> >> >> > IIS
> >> >> > 6
> >> >> > /W2K3 **without URLScan**?
> >> >> >
> >> >> > I've found references for doing so on other platfroms, particularly
> >> >> > Apache,
> >> >> > but so far none on IIS. Additionally, it seems that Java
> >> >> > useragents/bots
> >> >> > are
> >> >> > enough of a concern that even Google has recognized such and does
> >> >> > exactly
> >> >> > what we're attempting to do (denies requests).
> >> >> >
> >> >> > Thanks to anyone who can provide guidance on this. I hope that a
> >> >> > solution
> >> >> > is
> >> >> > available and can be shared with all IIS admins.
> >> >> >
> >> >> > -----------
> >> >> > Cheers,
> >> >> > Ed
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
>
>
>
.