Re: Crossover Error Rate (WAS "Intrusion Prevention")

From: Raistlin (raistlin@gioco.net)
Date: 12/11/02

  • Next message: Chris Petersen: "RE: Intrusion Prevention"
    From: "Raistlin" <raistlin@gioco.net>
    To: <focus-ids@securityfocus.com>
    Date: Wed, 11 Dec 2002 20:15:55 +0100
    
    

    > Just as with an IDS, you can reduce
    > one at the expense of increasing the other, but unlike IDS, there's a
    > commonly-known standard called the CER, or "Crossover Error Rate,"

    That's not indicative, really.

    In evaluating a system with that metric, you are supposing that both kind of
    errors are equally costly. They could not be (for example, in a biomedic
    system it is FAR better to have a false alarm than a false negative !).

    In addition it is not known, a priori, if the cost linearly scales. Having
    10 false positives a day can be acceptable, 100 false positives may be a bit
    more harassing (but not, necessarily, 10 times more), while of thousands of
    false positives are completely unmanageable (they have an "infinite" cost:
    we don't absolutely want to have that). At the same time, 1 false negative
    may be bad, and 100 false negatives are probably in the scale of "better to
    launch this crap out of the window".

    Please note that all the figures are totally subjective, and here only for
    the sake of an example, do not flame me on the figures :P

    What you really want to build is an ROC, Receiver Operating Curve, which is
    a diagram with a measure of the false positives on X axis, and a measure of
    the detection rate on the other. They are in some kind of 1/x - like
    relationship (the more false positives you accept, the better you find
    attacks, and vice versa). A "higher" graph (A larger area under it) means a
    "better" system, on the whole. But more accurately, you can match this
    graph with your own "cost function" for false detections and misses, by
    using really simple operational research techniques (you build the gradient
    on the graph, and find the tangent with the ROC curve).

    It's all theory with 40 years of background.

    Stefano