Re: IDS thoughts

From: Mike Frantzen (
Date: 05/20/03

  • Next message: Thomas H.Ptacek: "Re: IDS thoughts"
    Date: Tue, 20 May 2003 14:25:23 -0400
    To: Stefano Zanero <>

    > You are joking, right ? There's a whole lot of research still open in the
    > IDS field. Just to begin, you are apparently forgetting that there's a whole
    > paradigm of ID, anomaly-based detection, which has just been forgotten by
    > the mainstream development.

    I don't think anyone has forgotten anomaly-based detection. Most
    players are taking a hybrid approach.
    > In the next few years, while established IDS products will strive to keep up
    > to date their growing signature base, and face increasing performance
    > problems, probably some attention will be returned at that preliminary
    > choice of matching bad_things instead of good_ones.

    Keeping up isn't as hard as you would think. Even for pure pattern
    matching, most algorithms scale roughly on the order O(log n) w.r.t
    performance and number of signatures. Some of the poorer algorithms
    start getting ugly cache/tlb/bp behavior but there are algorithms that
    fix any two out of those three secondary affects.

    > When it comes to firewalling, we all agree: you just shut down everything
    > very tight, then open up what few ports you actually need. When it comes to
    > privileges and authentication, we do the same thing, and we are quick to
    > point out the error when someone tries to filter out unwanted input, instead
    > than specifying what is the expected one.
    > Oddly, when we talk about IDS and antivirus software, we blindly accept that
    > there's only one way to do it: by describing what we do NOT want on our
    > system by the mean of signature. Well, this happens to be a BAD idea, even
    > if until now it has given us some satisfactions.

    Ok. I do both firewall development (OpenBSD) and IDS development (NFR).
    And they are totally different, dare I use a buzz word, paradigms. To
    give a fairly deterministic measure of the difference, the size of NFR's
    regression tests alone are four times larger than the entire OpenBSD PF
    firewall. With firewalls, you have the luxury of being able to assume
    port == protocol (yes yes, you can do very rudimentary protocol
    validation with things like inspect scripting; and there are a few
    proxying firewalls that checkpoint hasn't killed off yet)

    That naivety from the security device just doesn't cut it in the IDS
    space. Sit down and stare at several captures of HTTP transactions.
    Ones from IE, Netscape, Konq, Opera.... They all look different and
    this is where theory diverges from implementation. An anomaly in one is
    perfectly normal in the other. It gets worse, the transactions start
    looking differently depending on the server they talk to. Sure, so we
    can leave the client protocol models agnostic of the server type. But
    now you start to factor in that HTTP clients lie about what program they
    are. And certain venders are notorious for protocol variations between
    minor patch levels...

    Yes, pure anomaly detection is a very complex process and *VERY*
    difficult to create something that doesn't have a propensity towards
    false positives. Subtle changes in an environments usage patterns often
    occur suddenly and tend to come across as anomalies -- now the IDS/IPS
    admin gets inundated after acclimating to relatively few alerts.

    Thus you see many venders transitioning (have already done so) to doing
    anomaly detection where feasible, and "bad thing" detection when not.

    I'll make a standing offer, I will buy anyone a cookie that can describe
    their enterprise network usage adequately enough that would allow pure
    anomaly detction. Hint, you control all the clients versions and all
    their connections either terminate at your servers or your proxies.
    frantzen@( | |
    PGP: CC A4 E2 E8 0C F8 42 F0 BC 26 85 5B 6F 9E ED 28


    IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities
    - including intrusion identification, relevancy, direction, impact and analysis
    - enabling a path to prevention.

    Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at:

  • Next message: Thomas H.Ptacek: "Re: IDS thoughts"