RE: Crying wolf: False alarms hide attacks : Eight IDSs fail to impress during the month long test on a production network.

From: Andrew Plato (aplato@anitian.com)
Date: 06/28/02 

Date: Fri, 28 Jun 2002 11:23:23 -0700
From: "Andrew Plato" <aplato@anitian.com>
To: "Graham, Robert (ISS Atlanta)" <rgraham@iss.net>, <focus-ids@securityfocus.com>

Robert Graham said...

> > Next time they should do RealSecure on one of my Win2k
> > appliances.
>
> No.
>
> While it is true that the reviewer found a bug with the Nokia
> platform that
> doesn't exist on Windows or Solaris, there wasn't anything
> especially wrong with the platform.

Yeah, but they should still buy an appliance from me just so I can make money and help them tune that puppy. :-)

> The issue is that the reviewer was hostile towards IDSs. A
> customer wants
> his product to work, so when they don't, they will keep
> calling tech support
> until it does. Reviewers want the products not to work, so they will
> construct the nature of the test in order to make sure this
> happens. The
> reviewer, in this case, never called ISS; the first we heard
> about him was
> at the end of this review, not at the first crash of the Nokia box.

I think another issue, one that Tom D'Aquino pointed out is that the reviewers had a rather inane concept of what an "false-positive" should or should not be. They assumed that if the IDS reported an event, but the target was not actually compromised, that this constituted a false-positive. This is patently incorrect since the purpose of an IDS is to detect intrusions regardless of their success.

> I'm not saying the review is wrong. As the reviewer said, he
> learned a lot
> about IDS during the process of reviewing these products. If
> you, too, don't
> know much about IDS but are planning to install one, you will
> likely get the
> same experience: being overwhelmed with alerts that are
> "false-positives",
> and a general sense that the product isn't working. The first
> few months of
> running the IDS are likely to be particularly frustrating. I
> suggest (a)
> working with a consultant to tune the system, (b) working
> with the vendor's
> support in order to get suggestions from them, (c) learning
> more about the
> system. You are going to do (c) anyway: after a few months,
> you are going to
> have learned a heck of a lot more about hacking and defense
> then you ever
> dreamed possible. Read the review: take it with a grain of
> salt knowing the
> reviewer wanted all the products to fail, but realize that
> this likely to be
> your experience the first few months after installing the
> product, you are
> likely to be overwhelmed with events and unlikely to be
> impressed during the
> first few months of ownership.

I think this is an area where all IDS vendors have missed the mark. They are not doing enough to educate users about the "post-installation blues" that can accompany an IDS. Many firms are rushing out to buy IDS solutions without really considering that that the process of implementing, tuning, and integrating an IDS can be very tricky.

I for one would like to see the IDS community start to focus more on the tuning process. At my firm we've begun to develop a "tuning methodology" to help our consultants (and users) get the most out of their IDS.

------------------------------------
Andrew Plato, CISSP
President / Principal Consultant
Anitian Corporation
http://www.anitian.com
------------------------------------