Re: Testing IDS with tcpreplay

Aaron Turner wrote:

1) Trying to do comparative analysis and you want to make sure each
device sees exactly the same thing

Hmm, why is that harder to accomplish with Metasploit than with tcpreplay?

Because metasploit, other tools and exploits incorporate PRNGs and
other methods of altering the attack so that it isn't exactly the same
each time.

Everything depends on defining what you want to test.

If you want to test the ability of detecting that exploit, you also want
to detect the ability of detecting its variations.

Therefore, you want to generate the attack in as many flavors and random
numbers as possible, and note down how many times each device catches
it. Then, the fact that these variations are "not the same" should get
less and less important as the number goes up.

If this still seems "unfair" to you, either you have very little faith
in IDS algorithms you are testing, or perhaps this is not what you
really want to test.

That makes "each device sees exactly the same thing"
really difficult.

Yes, but there is no reason to require that. It's a misconception of
"scientific repeatability".

Again, less complex (no 2nd box and vmware to maintain/automate)

This is surely true. However, how do you GENERATE that replayed stream ?

And once you have an architecture in place for generating it... why not
use it also for testing ?

Also what about attacks that Metasploit
doesn't have?

You run them yourself ?

What if you want background traffic?

You generate it ?

Your objections can be drawn exactly in the same way if you want to use
tcpreplay.

Unless you are suggesting to use some well-known-as-broken repositories
of data such as the DARPA datasets. You aren't proposing that, are you ?

If you're testing a vulnerable application then I agree. but if you
are testing an IDS/IPS, then I would argue that it is for all intents
and purposes it's the same thing. If you believe otherwise, then
please explain.

Some context-aware systems for instance would behave differently if an
exploit is executed onto a vulnerable application or onto a
non-vulnerable application. Anomaly detectors could not be able to catch
the attack itself, but may be very well able to see the consequences,
etcetera.

Say you have two IPS's you want to test. You an send an "attack" with
Metasploit against the first one and it detects it. You run it again
against the second one and it doesn't.

Consider this.

You send a tcp replayed stream against the first one, and then agains
the second one. The first one catches an attack, the second one doesn't.

Does this mean the first one is better ? No, it does not mean absolutely
anything.

The core of the problem is in WHAT YOU CALL TESTING, not in what you use
to throw packets at your IDS probe...

Regards,
Stefano

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------

Relevant Pages

  • Re: Testing IDS with tcpreplay
    ... why is that harder to accomplish with Metasploit than with tcpreplay? ... If you are testing you IDS you'd like to know that it accurately detects ... Also what about attacks that Metasploit ...
    (Focus-IDS)
  • Re: Testing IDS with tcpreplay
    ... IDS works if you use real attacks with real obfuscation techniques. ... Metasploit is a great tool for this. ... why is that harder to accomplish with Metasploit than with tcpreplay? ...
    (Focus-IDS)
  • Re: IDS Informer
    ... I would like to respond in kind to your message concerning IDS Informer. ... all but actually send the captured exploit by injecting the attack traffic ... >tcpreplay to record and replay the attacks onto a static wire later on, ...
    (Focus-IDS)
  • IDS Assessment (was: Intrusion Prevention... probably something else at one point)
    ... scrutiny of all IDS features/technologies. ... Anomaly-type detection engines can ... weaknesses of each detection methodology (which is described in much ... attack d'jour with a cool sounding name and/or press ...
    (Focus-IDS)
  • Re: Target based IDS review and discussion in Information Security
    ... This all began in 2000 when Marty lead the IDS development effort at ... > describes alerts as they pop out of IDS consoles. ... > Roesch names two other components as integral to target based NIDS: ... > an attack on a system that cannot succeed should be demoted. ...
    (Focus-IDS)