Re: Testing IDS with tcpreplay

No top post here...

On 2/23/06, Ivan Arce <ivan.arce@xxxxxxxxxxxxxxxx> wrote:
Hi Aaron.

See my comments inline as well ...

Aaron Turner wrote:

On 2/20/06, Ivan Arce <ivan.arce@xxxxxxxxxxxxxxxx> wrote:
Aaron Turner wrote:


If you are testing you IDS you'd like to know that it accurately detects
an exploit for a given vulnerability not that it detects the packets
that comprise a particular instance of the exploit's execution.

Well in both cases, you're only testing a particular instance of the
exploit. If you want to try 500 different instances of a particular
exploit you have to run metasploit 500 times. But when you test the
another IDS/IPS there is no guarantee that the next 500 times you try
it will look exactly like the first 500 times. Using
tcpdump/tcpreplay you could capture those 500 tests and replay them
any number of times, thereby making sure that all devices see the same
thing which provides true comparative analysis.

What I tried to point out is that with tcpreplay you end up testing
different things than with Metasploit or similar tools. Using tcpreplay
it good to build a valid and easily repeatable set of tests but it's
important to know the difference with the real thing lest you'll get
burned when a different exploit for the same bug or even the same
exploit in a different operational environment hits your IDS.

As long as you understand the limits of the tools, and follow a
methodical approach, then you shouldn't get burned. Don't get me
wrong, Metasploit is an awesome tool and belongs in everyone's
toolbox. IMHO, it's much better at pen tests and testing targets
(where the goal is to "just get in") then doing comparative analysis
of IDS-like products. Of course, replay tools NEED things tools
Metasploit to have something interesting to send in the first place.


3) Already have a library of pcap's (either from customers, the wild
or capturing traffic of real tools like Metasploit)
Yeah, but that is an entirely different kind of testing. Replaying the
packets captured from the execution of an exploit is not the same as
executing the same exploit again.

If you're testing a vulnerable application then I agree. but if you
are testing an IDS/IPS, then I would argue that it is for all intents
and purposes it's the same thing. If you believe otherwise, then
please explain.

Ok, I think I've touched on the point in my above comment but I will
explain my view a bit more.

I would argue that you are testing the IDS to figure out if it will be
capable of detecting exploitation attempts against real applications in
their real operational environment. If that is in fact the case, then
you need to make sure that the exploit traffic you're using to test your
IDS is likely the same that the IDS will see coming after the real
applications. It follows then that if the exploit that generates the
traffic will behave differently in the face of the real application then
your captured-replayed traffic can't really assure that the IDS will
work properly (unless you really understand how the exploit will behave
and what traffic it will generate in any random execution).

I agree with this in one way:

If your goal is to test if the one or more IDS will detect the exploit
against a SPECIFIC target (ie, the hosts it will be deployed to
protect) then yes, replay tools suck. But if you want to know if in
general one or more IDS's can detect an exploit against an arbitrary
instance of the target then replay makes sense. The actual exploit
tool will work to of course, but introduces unknown variables; like
does the target behave differently the second or third time? Of
course if you're not doing a regression or comparative test, then
using the actual tool provides everything you need.

My point is that modern exploits do behave differently based on the
runtime characteristics of the target system and application. I am not
talking about shellcode obfuscation or NIDS evasion attempts but about
exploit logic. For example if a given exploit needs to fill-up the
process heap after/before triggering the bug or obtain heap layout data
to change execution flow on the target app. then it will send more/less
-or even different- traffic depending on the endpoint's application state..

Yep. Which is why you should capture the same exploit being used
multiple times against multiple targets and then replay them.
Otherwise you won't know if each IDS saw each of the different
exploitation methods. Of course if you're not doing a comparative or
regression test, then you might not care (actually you might not care
if you are doing a comparative or regression test either, but I would
argue that you should :).

The point being you have to put forth the effort to automate
vmware/metasploit is a lot more work then just tcpreplay. Not to
mention, if you want to add other tools other then metasploit now you
have more automation work to do. tcpreplay is exploit/tool agnostic
and provides a simple and unified method of generating all kinds of
traffic- not just exploits.

agreed and understood, I am just pointing out that you will accomplish
different things depending on which test strategy you choose

Absolutely. Different tools for different needs/strategies.

Perhaps you're not aware, but tcpreplay fully emulates both client and
server state, from protocol setup, to the actual exploit and all the
way through getting a shell/whatever if that is what you captured. An

You mean at the TCP/IP stack level or does it emulate the application's
state as well?

Since tcpreplay sends both C->S and S->C traffic *through* the IPS (or
in front of an IDS) it replays the externally visible state of the
TCP/IP stack, OS and application. Obviously it doesn't track the
actual state of the target (by that I mean it doesn't track the stack,
heap or CPU registers) but if the attacker gets a shell and sends "ls
-l" to the target, the target will respond with a listing of the cwd
contents because that is what was originally captured.

The point being that the IDS/IPS doesn't actually track the state of
the target either, it only deduces the state by watching the traffic.
Tcpreplay sends traffic which is completely indistinguishable from the
"real thing".

Of course as we've discussed, there are variables which might alter
how an exploit happens between one attempt and another, in which case
you should capture/replay multiple executions of Metasploit/etc.

IPS/IDS has absolutely no way of knowing if traffic was generated by
tcpreplay or not, because it is EXACTLY like the traffic that was
originally generated/captured. If you want to run the same exploit
500 times with different parameters with Metasploit, that's great.

You can run the same exploit 500 times with exactly the same parameters
and the generated traffic will not be the same. Some examples:
- The Apache-SSLv2 exploit used by the Slapper worm
- An exploit for the php_memory_limit bug
- An exploit for MS License Logging Service (LLSRV)

The reason for this is that they have logic to take different decisions
or to use different data (pointers,etc) based on the heap state of the
target application.

Right. Which is why running Metasploit once isn't good enough to
answer the question, "Does this IDS always detect Slapper?". For
the same reason, you would have to capture/replay the same attack
multiple times. But at least when you did so with a replay tool you'd
be able to answer the question, "Do these IDS/IPS's detect Slapper in
all the same cases?"

Just capture each of those 500 attacks and replay each of them against
10 different IDS/IPS's.

The point is that with tcpreplay, your test cases are 100% repeatable.

Yes, but that does not assure 100% code coverage of the exploit you used
to generate the captured traffic.

Right now I'm not aware of any testing methodology that assures 100%
code coverage for an exploit.

As I said, I see the value of using tcpreplay but the implications need
to be clearly understood so you know what you're testing.

Agreed, but that's true of any tool.

Aaron Turner

Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to
to learn more.