Re: [fw-wiz] VPN endpoints

From: Paul D. Robertson (paul_at_compuwar.net)
Date: 09/01/04

  • Next message: Al Cooper: "[fw-wiz] Cisco VPN Client Behind a Cisco PIX or Router" 
    To: "Marcus J. Ranum" <mjr@ranum.com>
Date: Wed, 1 Sep 2004 07:41:40 -0400 (EDT)

    On Mon, 30 Aug 2004, Marcus J. Ranum wrote:

    > or the CIO magazine survey on security) - a lot of these surveys are
    > fundamentally flawed. They yield results but it's hard to say what the
    > results actually _measured_.

    So long as they're flawed approximately the same way from survey to
    survey, they're often both "better than nothing[1]" and a good relative
    metric. We often don't need absolute metrics, relative metrics will do
    just fine. I know what my $foo risk was last year, and I know what it was
    the year before, and I can compare to the survey and see the relative
    differences and the relative change- therefore, I can figure out my
    approximate relative change for this year.

    > Specifically, many security surveys are based on self-selected
    > samples (e.g: "polls"). When you do a poll, what you're doing is
    > asking "Please fill this out." But there are a lot of assumptions
    > that get dropped on the floor. :( What you're really measuring is:
    > - How much the person cared about the topic (motive to respond)
    > - How honest the respondent is (hard to verify)
    > - Other factors (hard to predict)

    You can also (a) drop outliers, (b) have cross-conflicting questions, and
    (c) answer the questions on behalf of a known quantity and still be able
    to validate polls pretty well. You obviously don't get people who don't
    care to respond, but if the number of people who do respond is
    significant, that's ok.

    > I'm sure nobody on this list has ever filled out one of those surveys
    > from a magazine in which they asked you your job position, whether
    > you were a decision-maker, company size, etc... And I'm sure you
    > all fill them out EXACTLY right. I used to enjoy periodically asserting
    > that I was the CEO of a 1 person company with a $4,000,000 IT
    > budget (well, a guy can dream, huh?) Unfortunately, sometimes

    You're out of the range of the mean by orders of magnitude, anyone doing
    it even half-way should be throwing that response away (assuming they
    *want* correct data,) which in that case is only half-right- better
    qualified leads should be worth more, but either fudging is built into the
    pricing model, you got sold cheaper, they didn't care, or someone got
    ripped off.

    Paul
    [1] That doesn't mean they aren't often worse than nothing, just that they
    can be useful. Just like assessing risk actually- same rules apply.
    -----------------------------------------------------------------------------
    Paul D. Robertson "My statements in this message are personal opinions
    paul@compuwar.net which may have no basis whatsoever in fact."
    probertson@trusecure.com Director of Risk Assessment TruSecure Corporation
    _______________________________________________
    firewall-wizards mailing list
    firewall-wizards@honor.icsalabs.com
    http://honor.icsalabs.com/mailman/listinfo/firewall-wizards

  • Next message: Al Cooper: "[fw-wiz] Cisco VPN Client Behind a Cisco PIX or Router"

    Relevant Pages

    • Re: Risk metrics
      ... We have updated this in OSSTMM 3.0. ... The OSSTMM has pulled out of RISK completely because it is so biased ... New metrics are quantification-based-- facts only from operations used ... > Vulnerability scans and pen tests are a snapshot. ...
      (Pen-Test)
    • Re: [fw-wiz] Re: Flawed Surveys [was: VPN endpoints]
      ... >>So long as they're flawed approximately the same way from survey to ... >>We often don't need absolute metrics, ... > I am sorry, Paul - if you believe the statement you made above, you ... > risk requires real metrics and statistics. ...
      (Firewall-Wizards)
    • Re: Alternative to Kerr metric, (kst).
      ... ellipsoid and we conclude the Earth is rotating, ... then a survey carried out on the surface most ... Look at the metric for a rotating frame. ... to say about metrics for rotating frames. ...
      (sci.physics.relativity)
    • Re: Alternative to Kerr metric, (kst).
      ... then a survey carried out on the ... I used the word "survey" carefully, i.e. a surveyor of the ellipsoid ... Look at the metric for a rotating frame. ... whatsoever to say about metrics for rotating frames. ...
      (sci.physics.relativity)
    • Re: Metrics for automation ...
      ... experimented with creating robust metrics and then gaming them. ... One of the easiest mistakes to make is to count test cases or bugs. ... significance of that risk, the power of the test to reveal the bug, the ... Would we really assess the quality of a car by ...
      (comp.software.testing)