Re: [fw-wiz] Re: Flawed Surveys [was: VPN endpoints] (Paul D. Robertson)

From: Marcus J. Ranum (mjr_at_ranum.com)
Date: 09/02/04

  • Next message: Skander Ben Mansour: "[fw-wiz] Linux Firewall Distributions" 
    To: Abe Singer <abe@sdsc.edu>, firewall-wizards@honor.icsalabs.com
Date: Wed, 01 Sep 2004 21:22:27 -0400

    Abe Singer wrote:
    >How about instead of continuing the "my idea is less f*ck3d than
    >*your* idea, there be a more productive discussion of what some good
    >methodologies would be for identifying, collecting, and analysing data
    >to produce metrics.

    Well, that's all in a Stats 101 textbook, or any good book on
    testing methodologies and statistics. That's the whole point:
    there is no need to reinvent this particular wheel wrong. It's
    been done; it's taught in most social sciences and math
    curricula at virtually any university.

    Normally, I am not one to "appeal to authority" on an argument.
    I believe that 100+ years of experience with testing, statistics,
    and polling, however, is not something to take lightly. ;) So
    I recommend the Stats 101 texts as a good starting point
    which will probably remove the need for further discussion.

    >* If you are going to do a survey, how do you target/vet respondents?
    >What questions do you ask. What controls do you have in place?

    Read any Stats 101 or experimental methods textbook. The
    reference I posted earlier on research methods (ISBN: 0767421523)
    has an excellent overview of the process.

    [...etc...]
    All the things you ask are covered in any introductory texts
    on research and/or statistics. Really. We don't need to go into
    it here! :)

    >It *would* be really useful to have some truly meaningful measurements.
    >It could do a lot to reduce the amount of snake-oil and magic security dust
    >beings sold.

    YEAH!

    I think the main point everyone seems to want to ignore is the most
    important one I made in my original posting:
    It's NOT MUCH HARDER TO DO IT RIGHT - it just takes a little
    bit of learning and some willingness to not charge straight in and
    start calculating the standard deviation of some bullsh&t. There's
    that old chestnut about how Computer "Scientists" have to re-invent
    the wheel every time because they're a bunch of immature jerks.
    I guess what I am saying is that it *appears* in this case (modulo
    sampling bias!) to be true - rather than learn statistics from a book,
    *EVERY* *SINGLE* security-related survey I have ever seen
    has significant methodological flaws. Are you guys comfortable
    being part of an industry that is somewhere between "witch doctor"
    and "cargo cult" on the spectrum of intellectual integrity?? I'm not!

    mjr.

    _______________________________________________
    firewall-wizards mailing list
    firewall-wizards@honor.icsalabs.com
    http://honor.icsalabs.com/mailman/listinfo/firewall-wizards

  • Next message: Skander Ben Mansour: "[fw-wiz] Linux Firewall Distributions"