Re: mier

From: Kurt Seifried (bugtraq@seifried.org)
Date: 09/07/01


Message-ID: <000901c13735$236e2560$6400030a@seifried.org>
From: "Kurt Seifried" <bugtraq@seifried.org>
To: <focus-ids@securityfocus.com>
Subject: Re: mier
Date: Thu, 6 Sep 2001 18:36:30 -0600

Weird, wrote an article about this exact topic several months ago.

=================

Truth in advertising
by Kurt Seifried

January 8, 2001 - In the computer industry the number of products released
every day, let alone every year is staggering. Along with these products we
get PR releases, extolling the many wonders and benefits to be gained should
we purchase them. Of course most of us in the industry have become somewhat
numbed by these incredibly cheerful messages that companies are more then
willing to share with us. Like many other industries a number of "testing"
laboratories have spring up, they test products and issue reports, giving us
the real facts, or so they claim.

Mier communications does many types of reports, however it would appear a
significant portion of their work ("NetWORKS As Advertised") are reports
commissioned by companies for free public distribution. While the promise on
Mier communication's webpage states:

Your customers know that your product has passed rigorous independent
testing in the Mier Communications Lab. They know your product really
networks as advertised.
Your sales team has Mier Communication's "seal of approval" to lean on when
they face tough customers who want proof your product works.
The NetWORKS As Advertised "seal of approval" and framed award are valuable
testimonials and marketing tools for your company.
Mier Communications provides you with a complete report of your product's
strengths and weaknesses, based on the testing.
It saves you the time, hassle and expense of setting up a lab and performing
the tests. We have the lab-and the experienced engineers at MierComm.
the actual content of the reports is quite different. For my first example
we will look at a recent report commissioned by NetworkICE on their BlackICE
Sentry IDS product, available at:

http://www.networkice.com/docs/Mier_Report_S.PDF

The report compares BlackICE Sentry, ISS RealSecure and Axent NetProwler. Of
these BlackICE is the only product capable of detecting +90% of attacks,
+90% of attacks when fragrouter was used, 100% network usage and it never
crashed. According to the report RealSecure and Axent are only capable of
60% and 80% detection of attacks at 20% of network utilization, as the
traffic increases BlackICE stays pegged at 100% and BlackICE and Axent
appear to fail miserable at 100% network utilization with less then 10% of
attacks detected. Of course with fragrouter being used to evade IDS
detection BlackICE is the only one to perform well, against detecting 100%
of attacks.

While I personally have not done any significant IDS testing personally I do
know someone who has (he is currently writing a report on various IDS
software packages for SecurityPortal), and I do know that the performance
while varied is not nearly as night and day as Mier would claim. Now if this
were the only such report I wouldn't be so worried, however Mier has dozens
of reports on various products.

The report on Indus Rivers is especially amusing to me. The report contains
essentially the exact same information that I was given in a PowerPoint
presentation by Indus Rivers on their (then) new product, I actually cannot
find any significant differences between the Mier "report" and Indus Rivers
marketing presentation on their product.

In general these Mier reports are extremely short (usually 4-6 pages, a
major portion of which is Mier contact info/graphics and so on). They do
explain the testing methodologies used to a minor degree however there are
often gaping holes. For example Mier states that the Intel 550T Routing
Switch was:

Rated "easiest to install and use" among Layer-3 routing switches tested by
Mier Communications.

This would be an interesting claim, but as far as I can tell the only
layer-3 equipment that Mier has tested is the Intel one, making the claim
somewhat pointless.

Take Consumer Reports for example, this organization has been testing
consumer products and issuing reports on them for several decades. The
methods and general practices used by Consumer Reports are exemplary, they
buy products from companies, as a normal consumer would, this prevents a
company from shipping them an improved product so as to do better on the
tests. As a rule of thumb Consumer Reports is not paid by companies to
generate reports on their products, and they do not even allow advertising
on their web site. Unfortunately most software testing companies do not have
such a stellar record, and there is a long line of product testing reports
with obvious biases or even severely flawed testing practices, such as the
Mindcraft report comparing Linux and NT.

As a reader of these reports is would be nice to implicitly trust that there
is no bias, however with reports being directly paid for by companies one
must always read them with a large grain of salt.

http://www.consumerreports.org/

http://www.mier.com/

Kurt Seifried, kurt@seifried.org
PGP Key ID: 0xAD56E574 Fingerprint:
A15B BEE5 B391 B9AD B0EF AEB0 AD63 0B4E AD56 E574
http://www.seifried.org/



Relevant Pages

  • Re: Letter to Lord Valve
    ... You have a difficult road ahead of you, with reports to ... deliver on email exchanges with hundreds of audio charlatans. ... advertising heap along with "Magic Audio Stones" and other nonsense, ...
    (alt.guitar.amps)
  • Re: Letter to Lord Valve
    ... You have a difficult road ahead of you, with reports to ... deliver on email exchanges with hundreds of audio charlatans. ... advertising heap along with "Magic Audio Stones" and other nonsense, ...
    (alt.guitar.amps)
  • Re: Advertising got thumped during the crunch
    ... True Blue wrote: ... reports the industry shed 180,000 jobs. ... Many of those jobs will never come back. ... Intelligent people have made spirited and almost complete defences of advertising through the years, but the fact remains it's not useful. ...
    (uk.politics.misc)
  • Re: Advertising got thumped during the crunch
    ... reports the industry shed 180,000 jobs. ... Many of those jobs will never come back. ... advertising through the years, but the fact remains it's not useful. ... Post-war was the era of mass ...
    (uk.politics.misc)