Re: [fw-wiz] Re: Flawed Surveys [was: VPN endpoints] (Paul D. Robertson)

From: Abe Singer (abe_at_sdsc.edu)
Date: 09/02/04

  • Next message: Melson, Paul: "RE: [fw-wiz] Wired article on the scientific method" 
    To: "Marcus J. Ranum" <mjr@ranum.com>
Date: Wed, 1 Sep 2004 19:13:03 -0700

    On Wed, Sep 01, 2004 at 09:22:27PM -0400, Marcus J. Ranum wrote:
    >
    > Abe Singer wrote:
    > >How about instead of continuing the "my idea is less f*ck3d than
    > >*your* idea, there be a more productive discussion of what some good
    > >methodologies would be for identifying, collecting, and analysing data
    > >to produce metrics.
    >
    > Well, that's all in a Stats 101 textbook, or any good book on
    > testing methodologies and statistics. That's the whole point:
    > there is no need to reinvent this particular wheel wrong. It's
    > been done; it's taught in most social sciences and math
    > curricula at virtually any university.

    I know it's all in Stats 101, what I meant was, assuming we've all read
    Stats 101, let's talk about these things in the context of computer
    security. How about some proposed sound methodologies for measuring of
    security thingamabobs?

    What are some hypotheses about computer security? How should we go about
    testing them? What kind of metrics would we like to see? Which methods
    in Stats 101 should we use to compute them? What data do we need to
    compute them? Where and how do we get the data?

    The answers are *not* in Stats 101, cuz the answers are specific to
    computer security. Other "sciences" have healthy debates over these
    questions wrt their own fields. Let's do the same!

    > >* If you are going to do a survey, how do you target/vet respondents?
    > >What questions do you ask. What controls do you have in place?
    >
    > Read any Stats 101 or experimental methods textbook. The
    > reference I posted earlier on research methods (ISBN: 0767421523)
    > has an excellent overview of the process.

    Again, I did not mean the general process, I meant in the context of
    computer security. What *specific* questions would you propose asking?
    How do you word them to get more accurate answers? Who specifically would
    you target for the survey? How do you go about getting a representative
    sample? What's the acceptible sample size? Etc.

    > All the things you ask are covered in any introductory texts
    > on research and/or statistics. Really. We don't need to go into
    > it here! :)

    No they're not, see above. :-)

    > >It *would* be really useful to have some truly meaningful measurements.
    > >It could do a lot to reduce the amount of snake-oil and magic security dust
    > >beings sold.
    >
    > YEAH!
    >
    > I think the main point everyone seems to want to ignore is the most
    > important one I made in my original posting:
    > It's NOT MUCH HARDER TO DO IT RIGHT - it just takes a little

    Well, in some cases it may be significantly more effort than sending
    out a bubble-form and saying "please mail this back to us, we'll be your
    best friend if you do, and we even included the stamp!"

    But, the other problem is that it's just not sexy or fun. A lot of this
    type of work is drudgery -- looking up data, putting it into tables,
    normalizing the data, doing some math, etc. Not nearly as much fun as
    building a skin an MP3 player, or yet another log parser, or setting up
    a blog on the web server, or installing linux on a c64...

    > bit of learning and some willingness to not charge straight in and
    > start calculating the standard deviation of some bullsh&t. There's
    > that old chestnut about how Computer "Scientists" have to re-invent
    > the wheel every time because they're a bunch of immature jerks.
    > I guess what I am saying is that it *appears* in this case (modulo
    > sampling bias!) to be true - rather than learn statistics from a book,
    > *EVERY* *SINGLE* security-related survey I have ever seen
    > has significant methodological flaws. Are you guys comfortable
    > being part of an industry that is somewhere between "witch doctor"
    > and "cargo cult" on the spectrum of intellectual integrity?? I'm not!

    This is basically where the medical profession was about 100 years ago.
    Medicines used to be hawked claiming to cure all sorts of ills with absolutely
    no clinical testing that showed any evidence of efficacy. Doctors could
    hang out shingles without any sort of license that showed a minimun of education,
    etc.

    Clinical testing, drug trials, medical licensing, use of medical histories
    and statistics has all developed in the last century. And it's not perfect,
    but it's a lot better than it was. We at least know that aspirin often helps
    relieve headaches.

    It's not that much longer that the notion of "scientific method" evolved.
    150-200 years ago it was not uncommon to make up data, "correct" data
    that didn't fit the theory, or throw out data that didn't fit.

    _______________________________________________
    firewall-wizards mailing list
    firewall-wizards@honor.icsalabs.com
    http://honor.icsalabs.com/mailman/listinfo/firewall-wizards

  • Next message: Melson, Paul: "RE: [fw-wiz] Wired article on the scientific method"