RE: Truth about False Positives

From: Jackie Chan (blue0ne@digitz.org)
Date: 09/10/01 

Date: Mon, 10 Sep 2001 14:22:02 -0400 (EDT)
From: Jackie Chan <blue0ne@digitz.org>
To: robert_david_graham <robert_david_graham@yahoo.com>
Subject: RE: Truth about False Positives
Message-ID: <Pine.LNX.4.33.0109101356500.1230-100000@bello>

I think we are missing the point here folks, regardless of what we want to
call the term, (personally I enjoy Shipley's terms such as "flying saucer
attack"), we need to solve the problem that is causing the term to even
come into existence. Perhaps a better term could be "meaningless drivel",
or "Junk alerts" Things that do nothing but clutter up the entire system,
from the sensor, to the network, to the console, to the database.

My first blame, is to blame the lack of education of the user when it
comes to deriving policies that are specific to their networks. Followed
by a close second blame to the developers who decide that any url with
'phf' in the string is suspicious.

So how do we solve this problem? First off you cant hand an erector set
to a child, and expect him to build something of his own creation without
spending time with him. Secondly, you can't build it for the child, and
expect him to learn from that... you have to teach him how to build, and
some of the laws of structure. Translated into business terms, this means
mandatory services, hopefuly good services. Transalted into tech terms,
this means that we have to figure out what it is we do when we ourselves
go through this process and automate that as much as possible...

(question: how many IDS developers have actually ever been responsible for
network security somewhere in the last 3 years? Now lets see how many
CTO's fit this same definition? Shockingly low, isnt it?)

Perhaps a recon scan of the network in question in order to profile it
should be our first step. And then we make take one sensor, and tell it
to spend its cycles doing all its wonderful stuff, wether signature or
proto based, finding things that would be bad for this network. Then we
take our second sensor, and let it be much broader, perhaps an inverse
policy... who knows.

We let our tier 1 folks watch for stuff from sensor 1, while our tier 3
analysts can compare that with oddities from sensor 2.

There are at least 5 different ways of tackling this problem of the
worthless information generated by our technologies, from origins of
development, to the level of effort of services, to a general business
model of sorts.

Regardless of how we attack the problem, lets stop quibbling over terms,
and mitigate the need for such a term.

-blue0ne

"The great bulk of my wealthy and educated friends regard me as a
dangerous crank."
- Theodore Roosevelt

On Mon, 10 Sep 2001, robert_david_graham wrote:

> Thanks for the idea of "benign triggers" (is that Cisco's exact
> terminology?)
>
> One of the zen issues is that IDSs don't detect intrusions, but instead
> detect things that CORRELATE with intrusions. For the most part, there is no
> such thing as a "false-positive" (unless there is a true bug), but there is
> a huge grey area between how well things correlate with intrusions. I like
> the term "benign triggers" much better than "false-positives", though we are
> likely stuck with the imperfect terminology.
>
> ISS is currently working on documenting the triggers for our signatures. We
> are "protocol-analysis" based rather than "signature"-based, so is not
> always a simple explanation as to why we trigger. Also, we are
> closed-source, so we aren't going to reveal to the customer the source code
> for them to figure it out.
>
> For an example, the following two "signatures" discuss how the product
> triggers, correctly or falsely:
> http://www.networkice.com/advice/intrusions/2001735/
> http://www.networkice.com/advice/intrusions/2000112/
> Scroll down to the "Trigger" section.
>
> Neither has a simple "pattern" that packets are scanned for, the situation
> is much more complex. We aren't going to document the exact procedure, but
> we are trying to give the customers some hints to work with. BTW: I picked
> those two because the "Trigger" help isn't all that extensive. Since we are
> going to have about 900 individual "signatures" to go back and document, the
> individual help will likely be a little sparse and not very helpful unless
> you understand the subject matter well. E.g. it is helpful to know that the
> Loki signature is more than just a single packet trigger, and that the
> Callit PING uses protocol-analysis rather than pattern-search. It'll get
> better over time, though.
>
> Note that there is a separate "noise" issue: yes, many events are real, but
> are they worth bothering about? A lot of IDS vendors are talking noise
> reduction these days. BlackICE correlates response codes with requests to
> detect success, RealSecure differentiates between scans and exploits (e.g.
> RealSecure detects more HTTP exploits than anybody, but triggers the least
> upon a CGI-scan), and many vendors correlate victim vulnerability/OS with
> attack. Some vendors are calling this "false-positive" reduction, but it
> really is a wholly separate class of problem. (It is an important problem,
> though).
>
> Rob.
> Lead Archtictect, ISS
>
>
>
> > -----Original Message-----
> > From: Greg Shipley [mailto:gshipley@neohapsis.com]
> > Sent: Sunday, September 09, 2001 7:33 PM
> > To: Klaus, Chris (ISSAtlanta)
> > Cc: 'issforum@iss.net'; 'focus-ids@securityfocus.com'
> > Subject: Re: Truth about False Positives
> >
> >
> >
> > On Thu, 6 Sep 2001, Klaus, Chris (ISSAtlanta) wrote:
> >
> > > If you know of any false positives, feel free to email me
> > with what is
> > > the false positives, what was triggering it, and any additional
> > > information you can supply, and we'll work to improve the
> > algorithm to
> > > remove the false positive.
> >
> > Something that Cisco (and others?) are doing, which ISS may want to
> > consider doing/expanding, is documenting "benign triggers" in
> > the product
> > itself. Personally, I find these very useful. For example,
> > if I see a
> > "IIS CodePurple Flying Saucer Attack" but there are notes
> > saying that this
> > is sometimes caused by the construction of a Krogarth over
> > the network in
> > the Total Annihilation video game, that saves me a lot of time.
> >
> > While it would be cool to completely eliminate all
> > false-positives, based
> > on my limited understanding of current signature-based
> > technology, I don't
> > forsee this happening (completely) in the near future.
> > Helping educate
> > the user with some useful data could help on this front...
> >
> > For whatever its worth,
> >
> > -Greg
>
>
> _________________________________________________________
> Do You Yahoo!?
> Get your free @yahoo.com address at http://mail.yahoo.com
> 

-- 
-blue0ne
http://www.digitz.org

"The great bulk of my wealthy and educated friends regard me as a dangerous crank."
	- Theodore Roosevelt

Quantcast