RE: Intrusion Prevention requirements document

From: FinAckSyn (
Date: 11/08/05

  • Next message: Talisker: "RE: Intrusion Prevention requirements document -Apology"
    Date: Tue, 8 Nov 2005 11:07:37 +0000 (GMT)
    To: Arun Vishwanathan <>,,,

    Hi VT,

    I strongly believe that replay tools are NOT an
    effective way to test an IPS:

    1) Replay tools are an unfair way to compare vendors.
     There is no measure as to the speed at which an IPS
    vendor responded to the original vulnerability. These
    .pcap files have been around for months or even years,
    giving plenty of time for vendors to catch up and
    write signatures to stop them. It's all very well
    having an IPS that responds favorably to a replay
    tool, but if certain signatures took days, weeks or
    even months to write, then this is not a fair way to
    compare device A with device B.
    It can be difficult to get hold of, but try and get
    the 'time to respond' stats from your proposed IPS

    2) Replay tools do not test variants. Although the
    pcap content may reflect a specific exploit, what
    about all the other exploits that abuse the same
    vulnerability? For example, the 50+ odd variants of
    Blaster, Slammer or Code Red? These test tools
    usually include pcaps for one particular variant only.
     A good freeware tool to test variations is
    Metasploit, or if you have spare cash, Canvas is a
    worthwhile investment.

    3) Replay tools do not test ability of a device to
    withstand volume. When worm/virus outbreaks happen,
    you don't just get a single packet in .pcap fashion
    that comes in, trips a signature, and gets blocked.
    You usually get several million of them. This is
    where it is important to test any rate-based features
    of the device. Also make sure these rate-based
    features don't block valid traffic. Plenty of
    freeware tools available - Apache Benchmark, nmap,
    Nessus, hping2 and a SYN Flood tool called Juno.
    What's more, these tools do not rely on .pcaps, so are
    a lot more real world. Be warned that replaying an
    identical pcap several million times is not an
    accurate way to rate-test, seeming all L2/L3
    information is identical in each packet (eg no change
    in SEQ). Devices that are good at DDOS tend to be
    just as good at withstanding sudden, large propagation
    of worms.

    4) Replay tools do not test IPS avoidance. Well, you
    may get vendor-supplied pcaps like
    blaster_with_fragments.pcap, but there are so many
    other ways to evade an IPS, and can you be sure that
    the vendor supplied .pcap is a reflection of real
    world traffic? Get hold of fragroute, tcpsic, plus
    nessus has some good evasion options.

    5) Replay tools do not test zero-day protection. This
    is the fun bit. How do you test a network IPS
    protects you against the shape of things to come?
    If you have a research team that can generate
    signatures to protect you quickly, then great. But
    there's still a window of opportunity during which
    your security can be breached.
    If an IPS is firing back events such as Sasser.W32/A
    blocked, or Code.Red.B.W32/Z, then chances are, it's
    not an anomaly based system that will give you much in
    the way of zero-day protection. But if it's firing
    events like 'CIFS Field Too Long' or 'HTTP Header
    Contains Illegal Characters', then this is indicative
    of the machine having good anomaly/zero-day procetion,
    rather than specific signatures for specific
    pre-historic viral events... ;)

    IPS testing is a big task, and needs big tools - to do
    things properly, take a look at the NSS testing suite,
    for example -

    In conclusion (I hear sighs of relief...):

    * Replay tools cannot be solely relied on to test an
    effective IPS
    * Think about what you want your network IPS to do -
    content-based checks are important, but equally as
    important are access control and rate-based ability.
    * A network IPS will never provide 100% perimeter
    protection. Always invest in extra security layers,
    especially host-based, to ensure that anything the IPS
    lets through does not cause problems
    * If you buy an IPS on the merits of tcpreplay
    results, you risk being hit with a zero-day threat or
    DoS condition, and losing your job.
    * Treat any vendor that promotes testing with a replay
    tool with caution (I learnt the hard way..)

    Hope this helps,



    --- Arun Vishwanathan
    <> wrote:

    > Hi VT,
    > I have used IDSInformer myself for testing and it is
    > a very good
    > product. There is a similar free tool (but lacks
    > certain features)
    > called Tomahawk which was released by Tippingpoint
    > some time back.
    > (
    > The working of these tools is very simple. You have
    > to assign two
    > interfaces. The tools consider one interface as
    > "client" and other
    > interface as the "server". The PCAP can be easily
    > split into two parts,
    > client traffic and server traffic. Consider the
    > following simple packet
    > sequence (A and B are IP addresses).
    > 1. A -> B SYN (client)
    > 2. B -> A SYN-ACK (server)
    > 3. A -> B ACK (client)
    > Packet 1 is first sent out on client interface. The
    > packet is expected
    > to arrive on interface 2 within a certain timeout.
    > On receipt of packet
    > 1, packet 2 is sent out on interface 2. Then packet
    > 3 is sent out on
    > interface 1 on receipt of packet 2 and so on. They
    > make the IDS believe
    > that it is seeing a real traffic situation.
    > In informer, you can change the MAC, IPs, Sport,
    > Dport of the packets.
    > In tomahawk you can only change the IPs at present
    > but if you want to
    > you can easily modify the code as its very simple.
    > There is no need to
    > configure any networks on the interfaces etc. Infact
    > the IPs, MACs can
    > be spoofed because it really doesn't matter.
    > Tomahawk has one limitation that it cannot test a
    > Layer 3 device because
    > it lacks support for specifying the source gateway
    > MAC and Destination
    > gateway MAC. It can test only Layer 2 devices.
    > Informer can be used in
    > both L2 and L3 situations.
    > In my opinion, both tools are great. I have used and
    > am using both tools
    > extensively. Informer also has an evaluation
    > version. You can download
    > it and try for yourself. For both the tools very
    > little configuration is
    > required.
    > Hope I was able to clear some of your doubts.
    > Regards,
    > Arun
    > -----Original Message-----
    > From:
    > []
    > Sent: Sunday, November 06, 2005 6:11 AM
    > To:;
    > Cc: Tony Haywood;
    > Subject: RE: Intrusion Prevention requirements
    > document
    > This sounds like a very viable solution that will
    > allow for testing. I
    > assume that it replays both the stimulus and
    > response of any
    > conversation and does not "fingerprint" the packets
    > at any layer with
    > the host OS TCP/IP stack (e.g. change of window
    > size, TTL, etc)? Does
    > the product automatically adapt to replay source and
    > destination traffic
    > based upon reading a libpcap file or do you have to
    > configure the
    > networks per card?
    > Has anyone else used this or a similar product in
    > their testing or other
    > security product tests? What issues did you
    > encounter?
    > Thanks for the feedback,
    > -VT
    > > One of the ways that you could test safely is by
    > using something like
    > > Traffic IQ Pro or a similar product. It is a
    > stateful traffic replay
    > tool
    > > and can be used to test any inline or packet
    > monitoring device.
    > >
    > > The product uses two network cards and so the
    > library of over 700
    > normal and
    > > threat traffic files can be replayed statefully
    > without the need to
    > connect
    > > to a live target system. This allows for live
    > production systems to be
    > > testing for the correct configuration really
    > quickly and easily.
    > >
    > > I have been involved in working in this area for a
    > number of years now
    > and
    > > my previous company was Blade Software where I
    > developed IDS Informer
    > and
    > > Firewall Informer to provide similar testing
    > capabilities.
    > >
    > > Information on Traffic IQ Pro is available below
    > should you want to
    > take a
    > > look.
    > >
    > >
    > > Working with testing labs and a number of security
    > and networking
    > vendors
    > > has enabled Traffic IQ Pro to be a really useful
    > tool for anyone who
    > wants
    > > to check the configuration of their firewalls,
    > IPS, IDS, routers,
    > switches
    > > etc and see how those devices perform under
    > different scenarios.
    > >
    > > Tony
    > >
    > > Tony Haywood
    > >
    > >
    > >
    > > -----Original Message-----
    > > From:
    > []
    > > Sent: 29 October 2005 20:40
    > > To:
    > > Subject: Re: Intrusion Prevention requirements
    > document
    > >
    > > Another question for everyone,
    > > When you brought in each vendor for evaluation,
    > did you configure a
    > test
    > > network for them or did you use your production
    > network? My 1st
    > concern is
    > > keeping my job :o) If I test in production, I
    > could impact production
    > > traffic. If I don't test in production, how can I
    > best ensure that I
    > won't
    > > have problems with custom applictions, older IP
    > stacks which could be
    > an
    > > issue if RFC compliance checks are done, etc.
    > > The vendor answer is always, "don't turn on
    > blocking and just
    > monitor." Is
    > > that a reality? I'd like some testimonials to
    > this and some real
    > life
    > > instances of what has been done from unbiased
    > sources.
    > >
    > > Thanks,
    > >
    > > VT
    > >
    > >
    > > > All,
    > > >
    > > > I work on a team that manages signature and
    > behavioral based
    > intrusion
    > > > detection systems today. We have been tasked
    > with reviewing IPS (or
    > > > whatever vendor name acronym you prefer) in '06.
    > Our normal process
    === message truncated ===

    To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre.

    Test Your IDS

    Is your IDS deployed correctly?
    Find out quickly and easily by testing it
    with real-world attacks from CORE IMPACT.
    Go to
    to learn more.

  • Next message: Talisker: "RE: Intrusion Prevention requirements document -Apology"

    Relevant Pages

    • RE: Intrusion Prevention requirements document
      ... Using a single tool as the only means of vendor comparison is of course over ... Replay tools augment existing testing methodologies and can ... generation but only based on packet captures so that fails in your criteria ... An IPS should easily detect this and drop the ...
    • Top IPS vendors - please read for invitation to Network World review.
      ... Network World is doing a head-to-head comparison on IPS ... vendor and you're NOT Barbed Wire, Lancope, Intrusion, Cisco, NFR, ISS, ... Precisely Define and Implement Network Security and Performance Policies ...
    • Talisker Site Returns - Rate/Review IDS Now
      ... Our vendor neutral site has been providing salient detail on every single ... Network IPS ... Application IDS ... Network Taps ...
    • Re: Intrusion Prevention requirements document
      ... Some excellent points Matt - the key to using replay tools effectively is, ... You can always use tools like Metasploit to create your own PCAPs, ... you can be sure that every vendor has a signature for them. ... > There is no measure as to the speed at which an IPS ...
    • Re: Signatures taking down network
      ... signatures were created in that release. ... Vendor might have used different test environment ... the signature pack by configuring a dummy network on ... IPS vendor. ...