Re: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk

From: Don Parker (dparker_at_bridonsecurity.com)
Date: 11/09/04

  • Next message: ADT: "Re: Snort signature packet generator" 
    Date: Tue, 9 Nov 2004 10:23:48 -0800
To: ADT <synfinatic@gmail.com>, focus-ids@securityfocus.com, Paul Palmer <b.paul.palmer@gmail.com>
    ('binary' encoding is not supported, stored as-is) Well the short answer to the question of "how do you verify the IDS/IPS" is simply
    put; get it tested by someone other then the vendor. Certainly no one is calling
    any vendor a liar when it comes to their claims however I would prefer to see a
    third party attesting to a vendors claims. After all if you are going to invest a
    very large sum of money on these technologies it makes sense to invest some in
    professional 3rd party review.

    Cheers,

    Don

    --------------------------------------------------------------
    Don Parker, GCIA GCIH
    Intrusion Detection & Incident Handling Specialist
    Bridon Security & Training Services
    http://www.bridonsecurity.com
    voice: 1-613-302-2910
    --------------------------------------------------------------

    On Mon, 8 Nov 2004 21:01 , Paul Palmer <b.paul.palmer@gmail.com> sent:

    >There is another issue to consider with IPS testing that you do not
    >see with IDS testing. While you can verify the detection capability of
    >an IDS/IPS using these tools if the captures you use show a complete
    >exploit (and not just the packets that one vendor thinks are
    >interesting), it is much more difficult to verify the protection
    >responses of an IPS using such tools. Detecting is a simple matter of
    >verifying that a reasonable event triggers when you replay the
    >capture. However, IPS devices are active devices and work by modifying
    >the network traffic that flow through them in response to undesired
    >events. This produces a variety of challenges.
    >
    >First, once the IPS responds, the remainder of the packets replayed
    >are likely no longer an accurate reflection of what would have
    >happened with live traffic. For example, if the IPS drops packets,
    >does the capture show retransmissions at the key points? If not, how
    >do you know that naturally occuring retransmissions wouldn't get
    >forwarded? Does the tool compensate for this by synthesizing
    >retransmissions on "dropped" packets?
    >
    >What if the IPS inserts TCP RST packets ahead of the undesired packet
    >instead of dropping it? Does the tool account for this? How does the
    >tool know whether the RST would have been effective? Does the tool
    >understand the protocol stacks it is simulating to be able to answer
    >that question?
    >
    >What if the IPS doesn't respond by dropping packets? What if it
    >rewrites the attack out of the packet before forwarding it in some
    >cases (because simply dropping or resetting an SMTP connection tends
    >to make matters worse rather than better for example)? How does the
    >tool know whether this rewriting would be effective?
    >
    >Ultimately, you get into a situation of role-reversal. How do you test
    >these tools? Well you exercise them in "real world" scenarios to see
    >how they behave. That is, you place them in a test bed with an IPS.
    >You launch "attacks" and see what happens. If the test fails, you look
    >to see if the tool failed or the IPS failed. Very rapidly you get into
    >the situation where the IPS is used as the "test tool" for your test
    >tool. That is, the IPS measures and heavily influences "proper" tool
    >behavior. Tomahawk works well at testing Tippingpoint's products
    >because that is how it was tested. It is horrible at testing ISS
    >products (the vendor I happen to work for) because Tippingpoint has no
    >vested interest in investing time and energy to make sure that ISS,
    >Netscreen, McAfee, etc (I apologize to all those I left out) are
    >adequately measured by their tools. I do not see how placing the tool
    >in the public domain solves this problem.
    >
    >So who is going to invest $1,000,000 in IPS equipment from various
    >vendors to properly test this tool and make sure it gives every vendor
    >fair representation? Who gets to decide what is fair? The vendors? The
    >public domain developers? Do you want to be one of those developers? I
    >can just smell the defamation lawsuits from whichever vendors feel
    >they are not being fairly represented by your code changes...
    >
    >FYI, I work for a vendor. We have a tool very similar to Tomahawk. It
    >works well at testing our products. However, other vendor's products
    >do not stand up as well to our testing :)
    >
    >The only reliable means of testing IPS product effectiveness I have
    >discovered so far is live fire testing. Setup an isolated LAN. Place
    >vulnerable systems on one side of the IPS and launch attacks from the
    >other side. If the vulnerable system is compromised, the IPS failed. I
    >strongly recommend modifying all of the exploits to only DoS the
    >victim so that nothing more than a reboot is ever necessary to prepare
    >for the next test.
    >
    >Paul
    >
    >On Sat, 6 Nov 2004 13:16:02 -0800, ADT synfinatic@gmail.com> wrote:
    >> (thread is getting long, so just going to snip the whole thing,
    >> hopefully you kept a local copy)
    >>
    >> Hey Greg/Marty,
    >>
    >> I don't think anyone would argue that tcpreplay or tomahawk are
    >> written for performance
    >> testing of IDS or IPS. I'm sure some people do that, but both have
    >> rather limited use in that regards (you want to generate background
    >> traffic using *your* network's traffic). What tcpreplay and tomahawk
    >> do rather well is provide the means to safely reproduce malicious
    >> traffic for testing detection capabilities.
    >>
    >> Unlike "live tests", tcpreplay/tomahawk don't require people to
    >> distribute working exploit code
    >> or attack an actual host which due to the nature of exploits will
    >> likely have to be "fixed" in some
    >> manner. Unlike exploit code, there is no risk that a pcap will also
    >> re-format your harddrive or
    >> require you to install and configure a wide variety of operating
    >> systems and applications to
    >> attack.
    >>
    >> Of course, unlike a "live test" there is some trust involved that the
    >> pcap contains packets which
    >> are relevant for the test you are running. Wether or not this
    >> precludes using either tool for being
    >> used by someone evaluating an IDS/IPS probably depends on how much
    >> they trust the pcaps.
    >> For those people who don't want to trust pcaps and don't have the
    >> means to get a library of working exploits, I'm sure Blade will be
    >> more then happy to sell you IDS Informer (of course, now you have to
    >> trust Blade, so you're just shifting your trust).
    >>
    >> Of course if you already have a repository of valid pcaps (maybe
    >> something the OSVDB guys could do?) with known attacks, then using
    >> these tools probably make a lot of sense for certain kinds of tests.
    >>
    >> Aaron, the tcpreplay guy
    >>
    >> --
    >> http://synfin.net/
    >>
    >>
    >>
    >> --------------------------------------------------------------------------
    >> Test Your IDS
    >>
    >> Is your IDS deployed correctly?
    >> Find out quickly and easily by testing it with real-world attacks from
    >> CORE IMPACT.
    >> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
    >> to learn more.
    >> --------------------------------------------------------------------------
    >>
    >>
    >
    >--------------------------------------------------------------------------
    >Test Your IDS
    >
    >Is your IDS deployed correctly?
    >Find out quickly and easily by testing it with real-world attacks from
    >CORE IMPACT.
    >Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
    >to learn more.
    >--------------------------------------------------------------------------
    >

    --------------------------------------------------------------------------
    Test Your IDS

    Is your IDS deployed correctly?
    Find out quickly and easily by testing it with real-world attacks from
    CORE IMPACT.
    Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
    to learn more.
    --------------------------------------------------------------------------

  • Next message: ADT: "Re: Snort signature packet generator"

    Relevant Pages

    • Re: Terminology: Inline IDS, IPS and Application Layer Firewall
      ... In fact, I would say that in most cases, packets are _bridged_ across them. ... You simply stick it inline and it bridges the traffic while sniping/blocking "bad" traffic. ... As to the differences between the 3 terms you mention, let's first make the assumption that IPS refers to an inline IPS. ... Inline IDS could simply refer to an IDS system that gets it's traffic by sitting inline. ...
      (Focus-IDS)
    • Talisker Site Returns - Rate/Review IDS Now
      ... Our vendor neutral site has been providing salient detail on every single ... Network IPS ... Application IDS ... Network Taps ...
      (Focus-IDS)
    • Re: ROI on IDS/IPS products
      ... since an IPS is nothing more than an IDS that can drop traffic;-) ... By purchasing an IPS from a vendor and enabling even *some* of the signatures for blocking I have established that I trust my vendor and I trust the signature authors to write signatures that are good enough to block an exploit or an attempt to exploit a vulnerability. ...
      (Focus-IDS)
    • RE: Generating Traffic to Stress Test IDS
      ... Vendor "A" may state one thing, ... >at any one time before starting to drop packets aggresively. ... >the packet drop rate on the IDS? ... >Great stuff seeking new owners in Yahoo! ...
      (Focus-IDS)
    • Re: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk
      ... There is another issue to consider with IPS testing that you do not ... First, once the IPS responds, the remainder of the packets replayed ... You launch "attacks" and see what happens. ... > testing of IDS or IPS. ...
      (Focus-IDS)