Mier Communications

From: Rick Williams (rickwi@hotmail.com)
Date: 09/09/01 

From: "Rick Williams" <rickwi@hotmail.com>
To: focus-ids@securityfocus.com
Subject: Mier Communications
Date: Sun, 09 Sep 2001 17:49:54 
Message-ID: <F42M4n1hUOM9Ku7Uw8n0000d75b@hotmail.com>

OK....long time listener, first time caller.....

There has been a spate of "lab bashing" on this list recently, with Mier
taking the brunt of it.

Firstly, let me say that I have not particular love for Mier - I do not like
the way they presented the Intrusion.com results, portraying it as capable
of detecting 100% of attacks at 100% network load. Then when you read the
small print it turned out they were using 1514 byte packets. Well duh!

That said, Kurt and others seem to be ignoring one rather inportant point -
these guys are a business, and they have to make a living somehow. In other
words, someone has got to pay for the testing. This can be one of three
ways:

1. The end user/reader pays - all testing is carried out free of charge but
the end user pays $1500-2500 for a nice comprehensive comparative review.
Not a popular choice for most of us here.

2. The vendor pays via advertising - the route taken by magazines (whether
print or on-line, I shall refer to them as mags)

3. The vendor pays via a test fee - the model that Kurt seems to object to

Personally, I see no difference between 2 and 3. Mag publishers get all high
and mighty about their so called "independence", but they are probably more
in the vendors pockets - especially in today's climate of dwindling
readerships and dwindling advertising revenue - than the labs who take a
direct payment from the vendors.

In fact, as far as I am concerned, in terms of integrity, there is (or
should be) no difference between all 3 methods. Any lab that is prepared to
"sell a good review" to a vendor will be found out at some point and that is
the end of their business. It's not worth the risk - so I do not believe
that it happens!

Vendors pay for an independent report, warts and all. Yes, of course it is
intended as a marketing aid - why else would they bother? And surely there
is noting wrong with Mier presenting it in this light when it tries to sell
its services to a vendor?

Vendors hope they will do well, and most who put their product forward must
be fairly sure that they will do well - they will have done some in-house
testing after all and will be looking mainly for an independent "seal of
approval" at this point. This means that vendors also know when they are
likely to do badly, and I am always highly suspicious of large comparative
tests where one or two key vendors are unwilling to play - to me it means
that they have something to hide.

Some of them shout loudly about "oh we only cooperate with magazines on
group tests". Well that's just as bad. I have a friend who is a freelance
journalist for a number of highly respected trade press publications and he
has told me how much those publications pay him for his work. This means he
gets about half a day to a day to spend PER PRODUCT maximum in order to make
it cost effective. That's barely time to get it out of the box, never mind
test it properly. Savvy vendors with something to hide know that they can
often sail through magazine reviews because they know that the testing is
not that rigorous.

Case in point - see how often RealSecure gets good marks in magazine reviews
- because it has a flashy interface and is easy to install and run - perfect
for the busy freelancer who often marks it very highly. Only when you spend
time and money putting it through its paces do you realise that it sucks
dead bunnies through a small straw in terms of detection rates on loaded
networks.

The reason Kurt never sees a bad review or a failure from a lab like Mier is
presumably because they have a contract with clients that gives the client
the right to pull unfavourable results. This is common sense from the
vendors point of view - what is the point of a marketing document that says
its product sucks? But it still wants to know that it sucks. The fact that
Mier only gets to publicise its success stories does not, therefore, mean
that it is biased.

I am talking about one-off tests here, of course, I assume that in group
tests there is more chance of being able to report on both good and bad
products - I obviously don't know how their contracts are structured. So
yes, it could be argued that if you want to know how truly awful a product
is, labs like Mier are not going to be telling you - but again, that still
does not mean they are biased.

As an end user, I am sure you could hire them to do an independent test of
producs on your behalf to give you the REAL lowdown on the products you are
thinking of buying - but you don't really want to pay for that, do you?
No....you want it for nothing....

So let's stop knocking the independent labs, huh? At the end of the day,
they are still our (as end users) best source of bias-free information on
all kinds of networking and security products, and certainly the only source
that most of us can afford. And the only way that happens is if the vendors
pay for the testing!

I certainly like to use reports from labs as a way of drawing up an initial
short list of product that I then move on to test more thoroughly myself
prior to purchase. I recognise the reports are not perfect, but used
correctly they can save me a lot of time - and you get nothing for nothing
in this life.

Just my 2c worth

rick

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp

Quantcast