RE: Comparing the performance of two IDS products with different architectures

From: Gary Golomb (gee_two@yahoo.com)
Date: 10/24/01 

Message-ID: <20011024164544.5991.qmail@web13401.mail.yahoo.com>
Date: Wed, 24 Oct 2001 09:45:44 -0700 (PDT)
From: Gary Golomb <gee_two@yahoo.com>
Subject: RE: Comparing the performance of two IDS products with different architectures
To: robert_david_graham <robert_david_graham@yahoo.com>, iheagwarac@aol.com, focus-ids@securityfocus.com, gee_two@yahoo.com

Hi there all.

This post is in response to a few things. To begin with, the original question is excellent! A
discussion on the morality of comparing IDS technologies as opposed to the usual discussions as of
late (ie: should-be snort user mailing list questions, why IDS X is better than IDS Y, etc...) is
a welcome breath of fresh air.

The second reason is out of frustration of seeing people who SHOULD know better making posts to
this list that resemble fancy marketing campaigns. Over the past two months, I ca not count how
many posts there have been from certain people that do nothing to advance our community and
science; while at the same time fueling fear, uncertainty, and doubt in this fledgling industry.
These are things that marketing departments have made a science out of spreading, and this list
WAS one of the last safe-havens from it.

Disclaimer: I work for an IDS company. To say that my views probably don’t reflect the views of my
employer and visa-versa is likely to be the understatement of the year.

Comments inline...

--- robert_david_graham <robert_david_graham@yahoo.com> wrote:
> Date: Tue, 23 Oct 2001 15:03:07 -0400
>
> > -----Original Message-----
> > From: iheagwarac@aol.com [mailto:iheagwarac@aol.com]
> >
> > Does anyone know if there is justification to compare the
> > performance of IDS
> > products with pattern-matching technique with those IDS
> > products that use
> > the state-based (protocol analysis)technique in the same
> > network environments?
> >
> > What will be the strength of the arguments?
>

>
> [Again, not in the Telnet example that a username signature is only tested
> against the username field -- another demonstration of the first point that
> a packet is only tested for a signature when needed, and not when it isn't
> needed.]
>

An interesting point, “a packet is only tested for a signature when needed, and not when it isn't
needed.” We’ll come back to that...

> I assume you are asking for some of the theory behind IDS performance,
> comparing "state-based protocol-analysis" vs "pattern-search" techniques.
>
> There are two fundamental advantages that state-based protocol-analysis has
> over pattern-search in regards to performance:
> 1. more efficient processing of traffic
> 2. scales better as you add more signatures
>

< blah, blah, blah >

> The theory behind this is that a pattern-search system doesn't know the
> contents of the packets, and must search that packet for many different
> patterns. In contrast, a protocol-analysis system knows the contents of the
> packet, and only tests signatures that apply to those contents.

< blah, blah, blah >

> The second part of the theory is that for pattern-search systems, the more
> signatures you add to the system, the slower the system becomes. If you look
> in the documentation for the average sensor, it will have a comprehensive
> discussion on how to remove signatures in order to improve performance. This
> isn't applicable to a state-based protocol-analysis system.

The terms for describing what a particular IDS is doing just keep getting longer and longer. We
could argue all day long about the strengths and weaknesses of “pattern matching” vs “protocol
analysis” vs “anomaly detection” vs "state-based protocol-analysis".

The above paragraphs describe one way of doing ID. Don’t get me wrong either. No one claimed that
it is the “best” way to do ID, so I am not attacking that point.

I am challenging the belief that you can simply say these generic terms we attach to technologies
have any sort of weight in this type of discussion - the type of discussion the original email
brought up.

For example, a "state-based protocol-analysis" IDS needs to do a couple of things before it can
determine whether it needs to send an alarm. Like Robert said, it needs to 1) recognize the
protocol and extract the fields it will be doing an analysis on, and 2) ummmm... do pattern
matching? Well, unless the device has some sort of ESP, it has do some kind of comparison, while
seems like looking for patterns, which I in turn call pattern matching - even though that word was
avoided like the plague in the above email when discussing his own IDSs.

Now, looking at a generic silly old “pattern matching” IDS... A signature for most of these IDSs
will probably contain a string of bytes, and maybe even a couple of wild cards. So this IDS takes
in EVERY SINGLE PACKET and looks for DNS attacks or RPC attacks in them, right? No? Oh, so they
are doing some form of protocol-analysis right? I mean, how else do they know to look for DNS
attacks in DNS packets? And looking back at the Telnet login example - if your pattern matching
IDS is looking for any unprintable characters (ie: stings of metacharacters, shell code, etc.),
then isn’t it doing anomaly detection also? So they are, to a great extent, accomplishing the same
thing - just differently.

You bet it is which is why all of these marketing terms people inject into these discussions are
not only frustrating, they are EXTREMELY misleading.

So, does that mean all IDS are doing the same thing? Not whatsoever. Just like a "state-based
flex-capacitor protocol-analysis string-theory" IDS does not check for RPC attacks in an ICMP echo
request, neither does any other IDS. The difference is in the decision-making processes. The
algorithms, binary-search trees, hash tables, etc. - all the parts that actually do apply to
discussions about speed.

Then there is a discussion of actually alarming on attacks. The fastest binary-search tree with
the most advanced protocol decoder doesn’t buy you anything if your IDS still isn't even looking
for the “signature” of an attack in the first place. (I use that term loosely.)

> This doesn't come for free: the state-based protocol-analysis that knows
> whether or not it should test for signatures itself costs the same as
> testing for a couple of signatures. Thus, the per-packet cost for
> pattern-search might be 10 signatures, and the per-packet cost for
> state-based protocol analysis might be 2 signatures.

There is more of a cost than only that. This discussion is about speed, but I think it should be
stated what we mean when we say SPEED. We are not talking about how fast an IDS can generate an
alarm. If two equal IDSs are looking at the same packet and one has 10 signatures and the other
has 2, then of course the one with 2 signature will be sending out its alert a fraction of a
second faster.

But that is not what we are talking, nor do most people really care about. We are talking about
the IDSs ability to make it though all of its signatures it needs to match against and move onto
the next packet and start all over without dropping any packets. This is the point I think you are
making above, and simplifying it by comparing the costs in terms of signature count. This can
easily go in a couple of different directions though. If you can look at 10 signatures in the same
amount of time as another IDS looks at 2, wouldn’t you rather have the 10? “But wait,” you say, “9
of those signatures would not need to be looked at if more advanced protocol analysis was done up
front so my 2 signatures are better.” It's hard to not bring the question about some vendor's "Gig
IDS solution" at this point.

And this is where our thoughts differ greatly. I would rather be given too much insight into
packets than not enough. To assume that something is always going to be were it “should be” inside
of a packet is what got the computer industry into this whole security mess in the first place,
and why people get to write tools like Sidestep. When we assume that something is going to be
where it “should” be is a dangerous assumption and EXACTLY how most all remote exploits and
attacks work. For protocol analysis to work - the way you describe it - you need to analyze
patterns in the packets that indicate you are looking at the right field. IMHO, as an IDS
vendor/creator/etc to follow the same methodologies as the faulty application we are watching is a
flawed and jaded way of looking at things.

To be cut and dry about it, if I am looking for a telnet login attack of some sort, then I want to
look at ALL telnet traffic going to a server (if it’s a server side attack). If I know that this
attack will come in the first 30 bytes of a packet, then great, lets just look at the first 30
bytes and move on, but to assume that it will always have a certain preamble or postscript of
bytes (indicating it is enclosed in the “right” field) seems to me to be a dangerous game - the “a
packet is only tested for a signature when needed, and not when it isn't needed” theory. I would
love to see a discussion on here about “when it is needed.”

> This is the theory behind the comparison. In practice, there are a lot of
> issues that can become more important. For example, CPU speeds are doubling
> every year.
>
> Robert Graham
>
> Disclaimer: I'm the architect of BlackICE and RealSecure 7 intrusion
> detection systems from ISS,

So back to the original question... Is it fair to test IDSs of different technologies against each
other? I think it is. As an end user, why should I care what algorithm or type of hash table a
prospective IDS uses. While it is very interesting information to know, I am going to be more
concerned about if the IDS can do what their propaganda tells me it can and that it is going to
actually alert me to specific events taking place on my network. Unfortunately, there isn't much
in the way of integrity checking in this industry - and there are quite a few people out there who
use that to their advantage. I have heard of companies (some much more respectable than others)
that are working on changing that though. Just make sure you know what you are looking for and
what is important to you in “performance” testing. If a test keeps all things equal (ie: traffic
types over an extended about of time, AND the IDSs are configured to be looking for relatively the
same things), then of course it is fair to test. I'm sure there are many other people who can
[will] elaborate on that subject though... :)

Anyways, sorry for the long-winded response.

-Gary Golomb

"There is more information in one thimble of reality than could be understood by a galaxy of human
brains. It is beyond the human brain to understand the world and its environment, so the brain
compensates by creating illusions that act as replacement for understanding. When the illusions
work well and the human who subscribes to the illusions survives, those illusions are passed to
new generations. The human brain is an illusion generator."

-Scott Adams, from "God's Debris"

__________________________________________________
Do You Yahoo!?
Make a great connection at Yahoo! Personals.
http://personals.yahoo.com

Relevant Pages

  • Re: Value of "richer" signatures?
    ... Snort, Dragon, and NFR, and I can tell you that they ... Here's an example of how the newer IDS signatures help ... Let's say you are using a simple packet grepping IDS ... > an FTP connection). ...
    (Focus-IDS)
  • RE: Value of "richer" signatures?
    ... Is it that much faster to do "protocol parsing" than ... > Here's an example of how the newer IDS signatures help ... > Let's say you are using a simple packet grepping IDS ...
    (Focus-IDS)
  • RE: Testing IDS/IPS Signatures
    ... can a scanner be used to validate the IDS ... True, Nessus can help in testing signatures but IMHO, it has limitations. ... > service features in Nessus and NeWT to see what is in fact ...
    (Focus-IDS)
  • Re: How to choose an IDS/FW MSS provider
    ... > people's IDS technologies, their opaqueness drives a constant nagging ... not becuase your signatures are open. ... NFR is not a free ... >> Senior Systems Engineer ...
    (Focus-IDS)
  • RE: Intrusion Prevention requirements document
    ... The tools consider one interface as "client" and other ... Packet 1 is first sent out on client interface. ... > my previous company was Blade Software where I developed IDS Informer ... Up to 75% of cyber attacks are launched on shopping carts, ...
    (Pen-Test)