Re: Testing IDS with tcpreplay
- From: Ivan Arce <ivan.arce@xxxxxxxxxxxxxxxx>
- Date: Thu, 23 Feb 2006 18:16:55 -0300
See my comments inline as well ...
Aaron Turner wrote:
On 2/20/06, Ivan Arce <ivan.arce@xxxxxxxxxxxxxxxx> wrote:
Aaron Turner wrote:
Generally speaking, tcpreplay is better when one or more of theHmm, why is that harder to accomplish with Metasploit than with tcpreplay?
following is true:
1) Trying to do comparative analysis and you want to make sure each
device sees exactly the same thing
Because metasploit, other tools and exploits incorporate PRNGs and
other methods of altering the attack so that it isn't exactly the same
each time. Sure it's the same "exploit", but the bits on the wire
are different. That makes "each device sees exactly the same thing"
That was exactly my point.
If you are testing you IDS you'd like to know that it accurately detects
an exploit for a given vulnerability not that it detects the packets
that comprise a particular instance of the exploit's execution.
What I tried to point out is that with tcpreplay you end up testing
different things than with Metasploit or similar tools. Using tcpreplay
it good to build a valid and easily repeatable set of tests but it's
important to know the difference with the real thing lest you'll get
burned when a different exploit for the same bug or even the same
exploit in a different operational environment hits your IDS.
really difficult. Of course some tools allow you to control these
via seeds but others don't.
2) Need to automate or do a lot of regression testing and want asame as above....
stable and relatively simple lab environment
Again, less complex (no 2nd box and vmware to maintain/automate) and
100% repeatable data streams. Also what about attacks that Metasploit
doesn't have? What if you want background traffic?
Those are certainly good points.
Regarding attacks that Metasploit does not have; I guess then you'd need
to find an attack for the bug elsewhere (either publicly available or in
a commercial product), otherwise how would you have the replayable
attack traffic anyway?
As for background traffic, yep that is certainly helpful and I recognize
the value of tcpreplay there, which is way smarter than simply using a
traffic generator to inject random packets that are completely out of
context for the test environment
3) Already have a library of pcap's (either from customers, the wildYeah, but that is an entirely different kind of testing. Replaying the
or capturing traffic of real tools like Metasploit)
packets captured from the execution of an exploit is not the same as
executing the same exploit again.
If you're testing a vulnerable application then I agree. but if you
are testing an IDS/IPS, then I would argue that it is for all intents
and purposes it's the same thing. If you believe otherwise, then
Ok, I think I've touched on the point in my above comment but I will
explain my view a bit more.
I would argue that you are testing the IDS to figure out if it will be
capable of detecting exploitation attempts against real applications in
their real operational environment. If that is in fact the case, then
you need to make sure that the exploit traffic you're using to test your
IDS is likely the same that the IDS will see coming after the real
applications. It follows then that if the exploit that generates the
traffic will behave differently in the face of the real application then
your captured-replayed traffic can't really assure that the IDS will
work properly (unless you really understand how the exploit will behave
and what traffic it will generate in any random execution).
My point is that modern exploits do behave differently based on the
runtime characteristics of the target system and application. I am not
talking about shellcode obfuscation or NIDS evasion attempts but about
exploit logic. For example if a given exploit needs to fill-up the
process heap after/before triggering the bug or obtain heap layout data
to change execution flow on the target app. then it will send more/less
-or even different- traffic depending on the endpoint's application state.
The point being you have to put forth the effort to automate
vmware/metasploit is a lot more work then just tcpreplay. Not to
mention, if you want to add other tools other then metasploit now you
have more automation work to do. tcpreplay is exploit/tool agnostic
and provides a simple and unified method of generating all kinds of
traffic- not just exploits.
agreed and understood, I am just pointing out that you will accomplish
different things depending on which test strategy you choose
In general, tcpreplay isn't all that useful IMHO when you're firstAhh that's interesting, I see it in exactly the opposite way: tcpreplay
starting off and "want to do some IDS/IPS testing" or only intend to
run a few tests or tests only once or twice unless you already happen
to have a nice pcap library.
is ok when you want to scratch the surface of your IDS capabilities or
perhaps more appropriate for stress or throughput testing or very basic
regression testing. However, if you truly want to check if the IDS
recognized real attacks you need to test with real exploit runs not with
a replay of their captured traffic.
Why? What is the different between "real exploit runs" vs. "replaying
real exploit runs" from the viewpoint of an IDS/IPS/UTM/etc?
I think I explained this in the paragraphs above.
Perhaps you're not aware, but tcpreplay fully emulates both client and
server state, from protocol setup, to the actual exploit and all the
way through getting a shell/whatever if that is what you captured. An
You mean at the TCP/IP stack level or does it emulate the application's
state as well?
IPS/IDS has absolutely no way of knowing if traffic was generated by
tcpreplay or not, because it is EXACTLY like the traffic that was
originally generated/captured. If you want to run the same exploit
500 times with different parameters with Metasploit, that's great.
You can run the same exploit 500 times with exactly the same parameters
and the generated traffic will not be the same. Some examples:
- The Apache-SSLv2 exploit used by the Slapper worm
- An exploit for the php_memory_limit bug
- An exploit for MS License Logging Service (LLSRV)
The reason for this is that they have logic to take different decisions
or to use different data (pointers,etc) based on the heap state of the
Just capture each of those 500 attacks and replay each of them against
10 different IDS/IPS's.
The point is that with tcpreplay, your test cases are 100% repeatable.
Yes, but that does not assure 100% code coverage of the exploit you used
to generate the captured traffic.
As I said, I see the value of using tcpreplay but the implications need
to be clearly understood so you know what you're testing.
"Buy the ticket, take the ride" -HST
CORE SECURITY TECHNOLOGIES
PGP Fingerprint: C7A8 ED85 8D7B 9ADC 6836 B25D 207B E78E 2AD1 F65A
Test Your IDS
Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
- Prev by Date: useful real-life example of IDS/IPS
- Next by Date: The Domain Name Service as an IDS
- Previous by thread: Re: Testing IDS with tcpreplay
- Next by thread: IPS test machine