Re: ICSA Labs Network IPS Testing

Hi,

Having some experience in developing and testing IPS, I have my two
bits to add. Most IPS tests, like Stefano said, are tricky at best and
pointless at worst. I don't want to take any potshots at ICSA or
anyone else, but it is not simple for anyone to do an exhaustive test
of an IPS and that too with the same test plan for every IPS.

ICSA, to their credit, say that of all the vulnerabilities they will
only focus on remote server-side vulnerabilities and that too only
those that they (and other vendors) think will affect enterprises.
Fair enough. They don't care about client-side vulns, local vulns and
vulnerabilities in Shoutcast.

They test a particular subset (however small it may be) and certify
the IPS. So even if one buys an IPS that blocks all server side
attacks launched by ICSA, it does not mean that the server behind the
IPS is secure from remote attacks. Vendors and buyers need such
certifications so that it is easier to make a sale and deploy an IPS
respectively – after all, not everyone subscribes to focus-ids.

It would be reasonable to criticize ICSA if one finds out they are not
doing what they promise correctly. But if the criticism is for not
testing exhaustively, that seems excessive.

Cheers,
Rahul

On 12/5/07, Stefano Zanero <s.zanero@xxxxxxxxxxxxxxxx> wrote:
Hi, didn't mean to interfere in your ongoing flame, but:

IPS certification testing, I thought I ought to correct some misleading
information

Oh, good, let's see! You don't mind if instead of going through your
whitepapers I just use your own email as a source, right?

IPS certification testing program. The truth is that we do not "pick
specific attacks and say that you must block these."

That's wonderful to hear. So, what do you do instead?

provides coverage protection for all attacks targeting an evolving set
of medium-to-high severity vulnerabilities that we and a consortium of
15 network IPS vendors
(http://www.icsalabs.com/icsa/topic.php?tid=6a87$5813f3e2-37b77ee3$3b4a-
f1d4a32d) believe are relevant to enterprise end users.

So, you pick specific attacks (which are a snapshot of a set of
vulnerabilities that you + the tested vendors believe are relevant) and
say "you must block these", right ?

This seems exactly the same sentence that Joel posted, only a bit more
elaborate :)

And just to shoot another shot in the dead horse of IDPS testing,
testing MISUSE based detectors (as most IPS are) on "detection rate" is
pointless. Testing them on coverage is tricky at best, and does not
really provide any useful insight at all on IPS where (as Joel pointed
out) having 60k signatures instead of 30k does not really mean anything.

Oh, and on a side note:

a) is in no position to speak authoritatively about ICSA Labs network
IPS testing,

The sheer fact that someone is "in no position to speak" about your
tests means that your tests are lacking. If a test is properly
documented and scientific, everybody is in a position to speak about it.

In the particular case of Joel Snyder, who has been doing excellent
tests for a long time, I'd say he is in a particularly good position to
comment.

If this email sounds harsh, well, it is. I just don't like people
commenting AGAINST other people, instead than pointing out the specific
flaws in their posts.

Best,
Stefano

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw
to learn more.
------------------------------------------------------------------------



------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it
with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw
to learn more.
------------------------------------------------------------------------

Relevant Pages

  • Re: Nortel Contivity 2600
    ... I would still put the outside interface of the VPN device behind an in-line IPS ... otherwise you could still be vulnerable to DoS attacks (IKE ... >> for vulnerabilities ...
    (Pen-Test)
  • RE: NIPS Vendors explicit answer
    ... this is the only comprehensive independent IPS test that's been ... Make sure the product continues to block attacks when simple, ... Test the IPS like you would any other network element (switch, ... The other vendors waiting for my tests:) are Netscreen IDP,RealSecure ISS Proventia G200 and Network Associates NAI Intruvert 2600 series. ...
    (Focus-IDS)
  • RE: ICSA Labs Network IPS Testing
    ... server-side vulnerabilities found in enterprise software. ... they are interested in ensuring proper protection for attacks targeting ... if folks have questions about ICSA Labs Network IPS ...
    (Focus-IDS)
  • RE: False Positives with IntruVert
    ... Subject: False Positives with IntruVert ... a different statement than IPS is not functional or not worth time or money. ... prevent attacks, ... profiled the attacks (signature or anomaly or combination of both)) has ...
    (Focus-IDS)
  • Re: IPS/IDS behavior with ISIC/UDPSIC/TCPSIC/ICMPSIC traffic
    ... considered as an attack that need to be protected by IPS devices? ... ISIC generates many packets with different IP protocols. ... If you still see 100% CPU problem, you may like to check you log settings. ... with real-world attacks from CORE IMPACT. ...
    (Focus-IDS)
Quantcast