Re: host-based ids evaluation

From: Greg Shipley (gshipley@neohapsis.com)
Date: 08/15/02 

Date: Thu, 15 Aug 2002 12:11:28 -0500 (CDT)
From: Greg Shipley <gshipley@neohapsis.com>
To: raffael.marty@ch.pwcglobal.com

On Thu, 15 Aug 2002 raffael.marty@ch.pwcglobal.com wrote:

> I agree to some extent. But let me add the following:
>
> 1. If you customize the Nessus Scripts a bit (I wrote UNIX-Scripts which
> are doing that automatically), you can issue attacks that the IDS should
> trigger on.

Interesting, but I'm wondering, are they real overflow attempts? All of
them? And if not, which ones? (I haven't dug through all of the NASL
scripts, but I seem to recall that while some of them actually go through
exploitation steps, many do not...)

I'm a real stickler here because I think there is a big difference between
crafting something that the IDS "should trigger on" vs. running real
attacks. The former strikes me as a bit of a kludge, particularly if you
are, say, reverse-engineering snort sigs.

> 2. Make sure you absolutely understand what the Scanner is doing. Nessus is
> of great help, as you get the attack-scripts with it. Look at them and see
> what they do. As soon as you can make sure that the attack is going over
> the network, the IDS should alert.

Er, weren't we talking about host-based IDS here? Regardless, while I
agree it is important to know what your tool is doing (good point!), I
think it is also important not to forgot what the goal of the test is: to
see if the product detects someone trying to exploit a vulnerability.

Just to add to what Andrew, Toby and you have stated, people should note
that most of the vuln scanners these days don't actually exploit the
service in question. They may poke at it, or they may go a step farther,
but until they actually exploit it, IMHO, they aren't truly attacking
anything.

From a thread earlier this year:

--------------------------------

(see http://archives.neohapsis.com/archives/sf/ids/2002-q1/0081.html)

- Scanners vs. exploit code: go with exploit code, if possible. Anyone
using a scanner to check whether their NIDS works or not better be PRETTY
DARN SURE that their scanner is actually ATTEMPTING to exploit those
flaws. Most don't. Many scanners rely on banner grabbing/checking, and
other tricks that don't "look" the same as exploit code to a NIDS. When
possible, use real exploit code, real vulns, or packet replays of real
exploit code (against real vulnerable machines). Real is better. :)

--------------------------------

I guess what I'm suggesting here is in-line with what Toby suggested, in
that if you're going to go the scanner route, you ought to be pretty darn
sure of what it is doing, under the hood. Most of them don't really
attack anything, in the true sense of the term. Now, if you want to test
your HIDS/NIDS' ability to detect PROBES, well then heck! Fire Nessus up
and have at it! :)

> 3. The problem of running "real" attacks is to have a good repository of
> them. I didn't have one for my work and was left with Nessus, which turned
> out to be quite helpful. (Do you have a collection of attacks that you
> would share?)

Yeah, this is the big challenge. This is also why we've stuck to the
"old-school" method of compiling exploit code, and running them against
vulnerable services. It is the only way to be sure, IMHO.

For whatever its worth,

-Greg

Relevant Pages

  • RE: Intrusion Prevention
    ... Coverage what can it detect; this covers basic attacks, ... IDS purchase. ... While doing these implementations and while working in an IDS vendor I ... sometimes we're told that we cannot see the testing methodology upfront. ...
    (Focus-IDS)
  • RE: Changes in IDS Companies?
    ... This means you need a standard IDS sitting behind it/next to it watching the ... Things like port scans and DoS attacks ... >>> If people are running insecure web servers, ... > Pretty sad state of affairs, when people don't update their patches at ...
    (Focus-IDS)
  • RE: Best Method(s) for signature verification.
    ... on this list - and other IDS lists - for the means to test their IDS ... When I say we use IDS Informer for our signature recognition testing, ... should point out that we do NOT use all the default attacks! ... (IIS attacks run against Apache web servers on Unix - "real ...
    (Focus-IDS)
  • Re: How to choose an IDS/FW MSS provider
    ... First, "recording everything" is not what IDS's were EVER meant for, ... others can create "audit" trails of every web request, every mail, every ... >detect attacks by inspecting layer 3 headers for prohibited IP ... >facility with an IDS or IPS deployed. ...
    (Focus-IDS)
  • Re: Alarming (was protocol analysis)
    ... Obviously, there are different ways to "detect" attacks, but John uses the ... no one should ever "rely" on any IDS for our ... As for Johns Metaphor of the motion sensor vs the pressure sensor, ... toward Intrusion Prevention as opposed to just Intrusion Detection. ...
    (Focus-IDS)