Re: mier

From: Kurt Seifried (bugtraq@seifried.org)
Date: 09/07/01


Message-ID: <001701c13756$57ec3ea0$6400030a@seifried.org>
From: "Kurt Seifried" <bugtraq@seifried.org>
To: <focus-ids@securityfocus.com>
Subject: Re: mier
Date: Thu, 6 Sep 2001 22:34:12 -0600


> [snip]
> Like many other industries a number of "testing"
> laboratories have spring up
> [/snip]
>
> I'm assuming you don't mean us. Miercom has been testing products for
over
> 12 years.

Actually I did, that's why I mentioned Mier by name.

> First, for someone who has not done "any significant IDS testing
> personally", if you are going to dispute results and you don't have any
> personal experience to draw from, it would be nice if you could provide
some
> real evidence to dispute these results. The friend-of-a-friend type
> explanation doesn't carry much water. You could start by providing a link
> to the test results you mentioned.

We had commisioned Dragos Ruiu (you know, writes packet reassembly code for
snort, sort of knows his IDS stuff) to do an IDS report for us (round about
the end of 2000), and he spent a lot of time testing and abusing IDS's to
get some good data. His conclusions oddly enough were not so skewed.
Unfortunately because securityportal.com is dead I do not have a link to the
report (Dragos, care to share perhaps?).

> Ok, this comment bugs me the most. How is this example a "gaping hole" in
> the explanation of our test methodology? I'm not even going to go into
the
> leap of logic you made here. But worse, at the end of the first paragraph
> of the report you cited, we state "in which the Intel Express 550T Routing
> Switch was rated the 'easiest to install and use' among 7 Layer-3 switches
> recently evaluated by MierComm for a competitive evaluation published in
> Business Communications Review (October, 1998)." The article can be found
> here

You could have also said it was the prettiest, or had the fewest sharp
edges, or was the least likely to cause brain tumors. The point is if you
only test one of course you can say it's "the easiest to install and use"
and be telling the truth (technically).

Your testing methodology might be perfect, and as your site claims provides
vendors with a safe "beta test site" to receive an objective evaluation of
their new networking products". Uut you provide this "seal of approval"
program that companies pay directly for .... and oddly enough I cannot find
any mention of products that have failed to get the seal of approval and I
find it hard to believe nothing would fail.

> Kevin D Brown
> Miercom

-Kurt