Re: Generating Traffic to Stress Test IDS

From: Greg Shipley (gshipley@neohapsis.com)
Date: 01/25/02 

Date: Fri, 25 Jan 2002 00:47:09 -0600 (CST)
From: Greg Shipley <gshipley@neohapsis.com>
To: Chad Gough <chad131@yahoo.com>

On Thu, 24 Jan 2002, Chad Gough wrote:

> Does anyone know of any good tools that can generate alot of network
> traffic to see at what point an IDS starts dropping packets?

Ack - I tried to stay out of this thread but the amount of noise bouncing
around is going to flush me out from my hiding spot. I can't take it any
longer!!!!

:)

Ok, Chad, before I talk your ear off about all of my various opinionated
ramblings on the state of IDS testing/benchmarking, a *lot* of this stuff
has been discussed before on this list. You might want to check out
things like:

http://archives.neohapsis.com/archives/sf/ids/2001-q1/0268.html
http://archives.neohapsis.com/archives/sf/ids/2001-q3/0247.html
http://archives.neohapsis.com/archives/sf/ids/2001-q3/0000.html

...and the threads that "surround" those posts. (My posts above are,
well, obviously biased. *grin*)

A couple of points:

=============================

- It's important to distinguish between BACKGROUND traffic, and ATTACK
traffic. I noticed that you requested "tools that can generate alot of
network traffic" but I see that a lot of people started forwarding
suggestions like stick. While tools like stick certainly can be useful,
stick is not geared for generating "background" traffic - it's for
generating attack traffic. BIG DIFFERENCE. Blasting 90Mbps of attack
traffic at a NIDS has little value, IMHO. You need a combination of
background traffic, with attack traffic "injected" at certain intervals,
which leads me to:

=============================

- While tools that generate attack traffic (like stick, Hailstorm, etc.)
are definitely cool/needed, as someone who does a fair amount of NIDS
testing I've found that our largest challenges have been with generating
legitimate background traffic, not attack traffic. If I were tucked away
in a vendor QA lab somewhere and was tasked with making sure 1000+ NIDS
signatures worked correctly, I might be singing a different tune. But I'm
like you - I'm looking for thresholds and breaking points. If you're
after finding breaking points, you probably only need 5-10 attacks to
determine whether the IDS is failing or not. Pick those 5-10 wisely, but
the real trick IMHO is the background traffic, which leads me to:

=============================

- I've found that the state of "benchmarking tools" that operate above
Layer3 is pretty sad. A lot of the makers of the expensive hardware units
are just now getting their tools to do basic things like, oh, say, 3-way
handshakes, using TCP ISNs other then 1, having adjustable sessions
lengths, etc. This becomes more important when you put the words
"real-world" before the words "background traffic." There's a big
difference between things like *real* HTTP-compliant web traffic vs.
things like a few constructed packets with port numbers set at 80 and
garbage jammed into the payloads. In general, I've been fairly impressed
with the commercial products from CAW Networks (they do web load testers)
and NetIQ's Chariot (they use the IP stack of the agent's host, so the
traffic is legit up to layer 4). I'm less experienced on the open-source
side when it comes to GOOD traffic generators. There's a lot of them out
there, but most of them seem to fall over when you really start pushing
them.

Back to the commercial stuff - be careful when looking at these tools.
You'll get things from some of those vendors like "Yeah, sure! We do
Layer7! Full web!" "But what if I want my web session to be more then 8
packets, Mr. Benchmarking Vendor? What happens if I want to have SMTP
traffic? What happens if I want to change lengths?" "Uh, er, we can't do
that yet. Yes, we know that most web transactions really are more then 7
packets, total, but, we're, er, still working on this...."

In short, the more "real" your background traffic (real SMTP, FTP, HTTP,
IMAP, SSH, SMB, etc.) the more legitimate your tests are going to be, and
the closer you will be to finding the REAL breaking point of an IDS. Which
leads me to:

=============================

- Legitimate background traffic is important for a number of reasons, one
of them is due to the fact that not all NIDS sensors have the same engine
design "under the hood." *Most* NIDS engines are based on either a
general "packet grep" model (SNORT, Enterasys' Dragon, etc.) while others
have a more protocol-aware approach (Intrusion.com's SNP, ISS' BlackICE,
etc.) [Note: many of the packet-greppers are building in protocol
pre-processor engines /normalizers/whatchamacallits, so these distinctions
might blur over time.] Some NIDS vendors groan when I draw this analogy,
but it's kind of like comparing a stateful packet filtering firewall to a
proxy-based one - one model cares about the application layer, one
doesn't. (and one checks it, one doesn't)

ANYWAY, if you've got one NIDS that's checking protocol compliance and one
that's just packet grepp'ing, they are going to be affected differently by
different types of traffic. This is why many of us testing d00dz have
gotten upset in the past when organizations use silly traffic going over
silly ports - it reduces the legitimacy of the tests and can serve as
points of confusion. But I'll spare everyone from that soapbox speech.
:)

=============================

- Some tests we've done before, starting from oldest to newest (some of
them with synthetic traffic gen, some without):

http://www.networkcomputing.com/1010/1010r1.html
http://www.networkcomputing.com/1023/1023f1.html
http://www.networkcomputing.com/1122/1122f3.html
http://www.networkcomputing.com/1217/1217f1.html
http://www.networkcomputing.com/1217/1217f2.html

Great reading if you want to make fun of me for my early testing mistakes,
er, I mean efforts. :)

=============================

Some other random thoughts:

- Traffic replay: works ok if you're dealing with traffic loads of under a
100Mbps. But if you want to test gig, er, it gets messy.

- Scanners vs. exploit code: go with exploit code, if possible. Anyone
using a scanner to check whether their NIDS works or not better be PRETTY
DARN SURE that their scanner is actually ATTEMPTING to exploit those
flaws. Most don't. Many scanners rely on banner grabbing/checking, and
other tricks that don't "look" the same as exploit code to a NIDS. When
possible, use real exploit code, real vulns, or packet replays of real
exploit code (against real vulnerable machines). Real is better. :)

- fragrouter is good for testing some of the anti-evasion components of
your NIDS. It's also really good for giving the NIDS a work-out. (Want
to test frag queues?) Great tool (thanks Doug!), but it only plays into
part of what you'll probably want to do.

- Concerning packet dropping stats: I'll let someone like Marty or Elliot
take this one on for more/accurate details, but packet dropping stats have
always been a problem. In order for them to be accurate you have to
assume that a) the NIC keeps those stats, b) the NIC can report those
stats *accurately*, c) the driver knows how to get those stats from the
NIC and report it to, d) the kernel, that may or may not present an easy
or accurate way for e) the user, you, to get that info. I've heard horror
stories about hard-coded settings of "0" being found in some driver code,
and that's just the start.

So in short, who bloody knows on that one. What I've always tried to do
is double-check things by counting stats on the sending machines, the
switches, and the receiving machines. Not pretty, or fun.

=============================

I'm hoping that doing good threshold testing gets easier in the future,
and we (Neohapsis) are trying to get some tools off the ground that will
aid organizations in their efforts. [Note: we gave up on the
Spirent/Smartbits stuff I mentioned last summer.] But unfortunately the
guys at Neohapsis are working on this stuff in their "spare time" (read:
at a snails pace), and I haven't figured out if this is something you can
get grant money for or not. (The problems of being more geek then
shmoozer....)

We're also keeping a sharp eye on what the ClicktoSecure folks are working
on. We're hoping their next-generation stuff can create both attack AND
background traffic. We'll see!

I hope some of this helps,

-Greg

P.S. Sam, I don't want to hear any crap out of you for this being so long.
:)

Relevant Pages

  • RE: Generating Traffic to Stress Test IDS
    ... the NIDS packet drop rate is about 50%. ... questions is - at what drop rate can an IDS afford to experience before ... >>Balancer, like that from TopLayer Networks, will be required, especially ...
    (Focus-IDS)
  • RE: Generating Traffic to Stress Test IDS
    ... > Seems that at 60Mbps throughput, the NIDS packet drop rate is about ... My questions is - at what drop rate can an IDS afford to ... are you doing any tuning of your NIDS? ... relying on seeing that one packet for a match is relying on too ...
    (Focus-IDS)
  • Re: ASIC-based vs. Software-based Security Platform
    ... With the emergence of network processors and the FPGA ... >>and the future direction of IDS. ... I can't say it's NIDS is as ... > new ASICs, however, there is a LOT of resistance to ...
    (Focus-IDS)
  • RE: IDS is out of context--was-->IDS is dead, etc
    ... and "buts" in accurately profiling your network, ... In those contexts, the risk is high and so is the impact. ... where fw and NIDS are run by different groups, ... a NIDS is not the security "solution" that they are marketed as. ...
    (Focus-IDS)
  • Re: Gartner is Dead, nCircle, Fusion, asset-correlation--was-->False positives, negatives and don
    ... > NIDS, correlate events, etc. etc. etc. ... locate heavily used servers on a network, but to date I don't know of anyone ... Security Event correlation with Vulnerability Posture and CAV. ...
    (Focus-IDS)