Re: IPS in the Enterprise UTM Firewall testing results

I wouldn't necessarily say that catch rates are disappointing. With IPS, it is very difficult to say what a good catch rate is. Clearly, the ISS box "caught" more things than all of the other guys, but remember that the purpose of an IPS is to handle that narrow window between problem and patch--if you are relying on your IPS to block SQL Slammer, you've got some major architectural conceptual errors in your network that IPS won't help you with.

I was pretty careful NOT to make any pejorative statement about the catch rate (except to say that relative catch rates give you relatively 'better' IPS), and I think that we ALL have to be careful in that area.

I don't believe that anyone can credibly put a stake in the ground and say "an IPS must block these specific attacks" and then defend that position. This is very different from, say, A/V or firewall, where there's a much clearer black-and-white line about what you need to support.

Clearly there are some pathological environments where an IPS somehow substitutes for a firewall and where 6000 signatures is the "right number" to have. But in enterprise deployments, it's very unclear to me how to adequately test an IPS for coverage. I can do performance easily enough, but checking coverage (which is what the Mu-4000 does) just seems quite dangerous.

Anyway, I think that it is useful to see the comparative values on IPS catch rate, but I would not go so far as to say that having an average catch rate in the 30% to 40% range is "bad" or "good" for these products.

I want to distance any testing we do from the bogus premise that you see in tests like the ICSA certifications where they pick specific attacks and say that you must block these. To me, that's not supportable. It may be in an IDS, but IDS and IPS are entirely different beasts, and we were testing these products as IPSes, not IDSes.

jms



Ravi Chunduru wrote:
this is really a great report and i am sure lot of effort has gone
into this. catch rates and perforamance is really caught my eye.

Catch rates are really disappointing across the board except for ISS.
i do understand that client attack detection is new, but even the
server side catch rates are awfully low. i understand that these are
expensive boxes. i did not see any vendor responses on low catch rate
and performace.

is this due to technology limitation or is it that devices tested are
not up to mark?

Ravi

On 14 Nov 2007 15:28:18 -0000, jms@xxxxxxxxx <jms@xxxxxxxxx> wrote:
After months and months and months in the lab, a huge UTM test I did for Network World is now available (for free, folks, for free) on their web site. I apologize in advance if you have to click 800 times to read the whole 19,000 words, but here goes:


Main story starting point:

http://www.networkworld.com/reviews/2007/111207-utm-firewall-test.html


Just the discussion of IPS in the UTM firewall/enterprise space:

http://www.networkworld.com/reviews/2007/111207-utm-firewall-test-ips.html


Chart on catch rates based on Mu-4000 testing:

http://www.networkworld.com/reviews/2007/111207ips.html


If you're not sure that enterprise should even be running IPS in their firewalls, you can click on the link below for a header page which has further links with some discussion on the pros and cons of that issue:

http://www.networkworld.com/buyersguides/guide.php?cat=865480


Enjoy or not, as you see fit.


jms


--

Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719

Senior Partner, Opus One Phone: +1 520 324 0494

jms@xxxxxxxxx http://www.opus1.com/jms


------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw
to learn more.
------------------------------------------------------------------------



--
Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Senior Partner, Opus One Phone: +1 520 324 0494
jms@xxxxxxxxx http://www.opus1.com/jms

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw to learn more.
------------------------------------------------------------------------

Relevant Pages

  • Re: IDS evaluations procedures
    ... an IPS when the attack was not a variant strain of a previous attack/virii ... like an IDS are they not? ... Traffic-based anomalies? ... Are you only interested in classic "attacks" (fire up Nessus, ...
    (Focus-IDS)
  • RE: Recent Gartner IDS/IPS report
    ... > resources to properly analyze security reports, ... > replace the IDS products. ... since these same vendors compete with your ... Basing IPS entirely on IDS and making the offspring a single product is ...
    (Focus-IDS)
  • Re: Fingerprinting IDS sensors?
    ... The nature of the response will tell you the nature of the defenses. ... That depends on if the IDS is monitored or not. ... An IPS will actively interfere with traffic patterns and you can find it by launching sample attacks at a target and watching for a response. ... With sufficient profiles of a set of IPS it would be possible to craft a tool that could identify which IPS is inline based on which attacks are blocked and how. ...
    (Focus-IDS)
  • RE: IDS evaluations procedures
    ... Tim might be making more emphasis on the protective nature of IPS, ... I disagree with the "real-world protection against ... IPS is helpful to stop, at least, known attacks, while requiring less ... attention than with IDS; on the other hand, IDS is helpful to detect a wider ...
    (Focus-IDS)
  • RE: IDS alerts / second - Correlation - Virtualization
    ... combinations that operating systems and applications respond improperly ... IDS alerts / second - Correlation - Virtualization ... any IPS has to do IDS first. ...
    (Focus-IDS)
Quantcast