Industry test reports... (was RE: [fw-wiz] Proverbial appliance vs software based firewall)

From: Marcus J. Ranum (mjr@ranum.com)
Date: 10/26/02 

To: "Ryan, Kennedy" <Kennedy.Ryan@viacom.com>, firewall-wizards@honor.icsalabs.com
From: "Marcus J. Ranum" <mjr@ranum.com>
Date: Sat Oct 26 19:40:16 2002

Ryan, Kennedy wrote:
>Funny thing.. I'm right now looking at a Netscreen commissioned Tolly Group
>test report from March 2002, (no.202121) which amazingly (not) shows
>Netscreen outperforming Checkpoint/Nokia IP740 and Cisco 545 by leaps and
>bounds..

Tolly group's tests are all bought and paid for. The vendor pays
to have a test of their own devising run, and Tolly reports the
results. Remember the big scandal from last year when Miercomm
published the results of their "test" of Intrusion.com's IDS?
The test was rigged (the protocol was suggested by Intrusion.com)
to yield completely ridiculous (and deliberately deceptive)
results. Typically, in a software company, it's the marketing
organization's budget that pays for the tests. :) Then the testing
company carefully words the results so that they don't step on
their other customers that have _also_ paid for tests.

The thing to look for is when they run the SAME TEST protocol
every time and make up their own test protocol with input from
outsiders not beholden to the vendors. I know I used to bash
them 8 years ago or so, but ICSA labs is doing some really good
stuff with their firewall tests these days. But they're looking
at functionality, not "marketing numbers" - don't believe _ANY_
of the marketing number tests.

When you buy a car do you believe the manufacturer's claim
that it goes 0-60 in a "blazing" 6 seconds, or Car and Driver's
test results where they used their standard test methods and
got 0-60 in 8 seconds? ;)

mjr. 

---
Marcus J. Ranum				http://www.ranum.com
Computer and Communications Security	mjr@ranum.com

Relevant Pages

  • Re: VW Losses in North America
    ... traditional marketing hype......Especially VWs.. ... > long, because their customers figured out, by owning them, that they are ... Because a rusty car in your, ... > with a new lineup, and starts their Very clever marketing campaigns, to ...
    (rec.autos.makers.vw.watercooled)
  • Re: Oil Peaked !
    ... The car didn't replace the horse ... I think it's Toyota that's marketing a hydrogen powered ... GM was going to do some test marketing on a hydrogen vehicle, ... Electricity is ...
    (misc.news.internet.discuss)
  • Re: Greedy Bastards.....
    ... you fell for the marketing bullshit. ... A production car. ... It's a ford that rolled off the ... Shelbyville Ford makes up for making $300 less per unit by selling ...
    (rec.autos.makers.ford.mustang)
  • Re: Oil Peaked !
    ... The car didn't replace the horse overnight, but eventually it did replace the horse. ... they intend on returning to Japan with the vehicle and then mass marketing it in the US. ... GM was going to do some test marketing on a hydrogen vehicle, and BP was going to build hydrogen stations as part of the marketing strategy. ...
    (misc.news.internet.discuss)
  • Re: Flash Loons
    ... The customer of an advertiser is a company's marketing ... that children influenced $9 billion worth of car sales in 1994. ...
    (comp.infosystems.www.authoring.html)