RE: [Full-disclosure] windows linux final study
From: security curmudgeon (jericho_at_attrition.org)
Date: Tue, 29 Mar 2005 17:06:26 -0500 (EST) To: "Lachniet, Mark" <firstname.lastname@example.org>
: Yes, but did you actually verify their research using their methodology
: to see if they screwed up?
Personally no, and I won't =)
: As in any study, the methodology and assumptions control the result. You
: can either poke holes in their methodology (for example by pointing out
: that the use of only published results is not a true indication of their
: security, in which case the eEye list of purported flaws is relevant) or
: you can use their exact methodology to recreate the work and prove that
: their data collection was wrong.
Exactly. Perhaps I should have expanded on that toward the end, but my
comment about mincing words refers to this. Like the previous Microsoft
funded report in 1999 , a company named Mindcraft set up a test to
compare Linux vs Windows NT Server 4.0 and judge their "performance"
(which was also rebutted fairly well ). The two things that Mindcraft
didn't mention at first were 1) Microsoft funded the test and 2)
Mindcraft's advertising completely destroyed their own credibility:
"With our custom performance testing service, we work with you to define
test goals. Then we put together the necessary tools and do the testing.
We report the results back to you in a form that satisfies the test
So like you said, if you design the methodology with the intent of
reaching certain results, you can't trust it as anything but a glorified
marketing brochure. Instead of pointing that out more clearly (as I
probably should have), I skipped past that to point out Microsoft is not
fast to patch. Of course they have a < 30 day response time when they
coordinate disclosure with these companies, many of which get advanced
copies of their software and don't want to burn bridges, and many of which
follow ethical disclosure (to a fault sometimes). In the real world,
discovery of a vulnerability isn't limited to one person or company. We
have seen this occur several times, where NGSS, eEye and iDefense 
discovered the same vulnerability, and the resulting advisory w/ patches
came out months later . During that time frame, we can't expect or
assume that no one else discovered the issue and/or used it for their own
ends. Focusing on the time between public disclosure and patch release is
a red herring (in this case).
: I guess what I'm saying is that you can't say the study is wrong if they
: release and follow their own methodology, but you CAN say its just plain
: not relevant due to the assumptions and methodology. And, lets not
: forget, the person who FUNDS the study probably CONTROLS the
In an ideal academic setting, I can't say the study is wrong if their
methodology is followed and results published. In reality, if I factor in
other things (like you just did with RedHat and 'official' patches), I can
say the study is wrong. There is often a serious disconnect between
academia and the rest of the world who actually deploys these platforms,
and that is lost in these 'studies'.
Full-Disclosure - We believe in it.
Hosted and sponsored by Secunia - http://secunia.com/