Re: Announcement: Alert Verification for Snort

From: Bill Royds (
Date: 10/25/03

  • Next message: Richard Bejtlich: "Re: Announcement: Alert Verification for Snort"
    To: "Martin Roesch" <>, "Stephen P. Berry" <>
    Date: Fri, 24 Oct 2003 23:30:31 -0400

    This is a very interesting point of view because it points to a new way of
    creating an IDS.
    What we really want is a language that describes the context of a system (in
    send of multiple servers, networks, programs etc.) and a language that
    describes the possible attacks at level of how those attacks can damage this
    system. We can then compile these two into a system vulnerability
    description that can optimally respond to data hitting the system.
       Now the main set of data for creating a particular implementation of an
    firewall is to have a systems description of data flows allowed into the
    system, a grammar of allowed input. This grammar can then create a filter
    that limits allowed data so that one has closer to an "allow only valid
    input, deny the rest" stance for actual data content, not just the present
    envelope view of most firewalls.
      The IDS' job is then to analyze that which was denied to determine why and
    tune the firewall to better filter in the future.
       The limit to this approach is similar to the concept of anomaly detection
    IDS, except that the anomalies are not strictly from context but from a
    grammar. The grammar is created by looking at the input/output descriptions
    for sostware running on the system (which includes the RFC for protocols
    allowed, the scanf/printf parameter set, "well known" values for buffer
    sizes in software etc.). This grammar view also allows for traffic
    normalizers attuned to the system that are protected.
      One integrates the concept of vulnerability scanner, firewall, IDS and
    honeypot in a single system.
    The vulnerability scanner and signature set create the grammar for the
    firewall rule.
    The firewall examines all traffic for conformance with the grammar. That
    which conforms is passed through to the system, that which fails is passed
    through to the honeypot that captures it and increases the contextual
    information for the IDS to analyze it (a honeypot running IIS would give the
    correct return code for an IIS attack without exposing the corporate data to
    the exploit). Essentially the honeypot mirrors the software base of the
    system without the vulnerable data set. True false positives (in Marty's
    sense) would point to a need for tuning the firewall to prevent a denial of
    service, but contextual false positives would only imply updating the
     This still has the risk of false negatives, but monitoring the reaction of
    the real system could add more context information to the grammar to reduce
    those as well.

    ----- Original Message -----
    From: "Stephen P. Berry" <>
    To: "Martin Roesch" <>
    Cc: "Christopher Kruegel" <>;
    <>; <>
    Sent: Thursday, October 23, 2003 10:33 PM
    Subject: Re: Announcement: Alert Verification for Snort

    Hash: SHA1

    Martin Roesch writes:

    > [...] I think it's important to make the distinction between "real" false
    > positives (i.e. the IDS screwed up) and nontextuals where the IDS has
    > done its job, it just needs more information to properly evaluate the
    > reality and priority of the event.

    Insufficient context is clearly -part- of the problem in many cases,
    and I don't really disagree with the above evaluation.

    I think the problem[0] is also more fundamental, however, and is based on
    a pervading infosec GCE[1] that holds that meaningful conclusions about
    large-scale behaviours (i.e., the remote compromise of a network-attached
    device) can be made on the basis of individual packets.

    This misconception isn't merely the result of misinterpreting
    the -scope- of an NIDS event (i.e., what a single signature match
    tells you), but rather the -character- of such events (i.e., what
    kind of information such events are capable of conveying).

    Some time ago while fiddling around with constructing a formal grammar
    for event handling for use in some future release shoki, a conceptual
    model occurred to me:

    NIDS signatures and most other intrusion detection heuristics (i.e.,
    regex searches through syslog output) merely serve as a mechanism for
    tokenising the raw data stream (packets on the wire or
    lines in /var/log/messages, for example). In other words they are
    components of a lexical analysis system, the output of which is a
    stream of recognised tokens.

    Currently, almost all NIDS systems do -just that-, and rely on the
    to actually parse these tokens.

    Expecting such a system to meaningfully predict the actual state of
    a network is kinda like grepping through a bunch of C code for
    things like `snprintf' and `fork', and expecting to be able to predict
    what the program will do.

    Evaluating whether a particular event emitted by a NIDS widget is
    a `false positive' is a semantic evaluation, not a lexical
    categorisation---but all most NIDS produce are lexical analyses. This isn't
    merely a problem of insufficient context---it's a misconception about the
    underlying character of the data.

    Does this make sense? I think this is a nontrivial distinction, but
    I'm not sure that the above makes it clear.

    That all aside, I further think that evaluating the content of a
    NIDS event is more complex than testing for the presence or absence of
    some quantity(-ies). I.e., is there an `actual' attack or not, and did
    the NIDS catch it or not.

    A rigourous discussion of this contention is probably beyond the scope
    of this message, but I would suggest that this evaluation is -at least as-
    complex as risk analysis, and is probably in fact more complex. Put
    a slightly different way: you can't really evaluate whether something
    is a `false positive' or not except within the context of a threat
    model. Since very few IDSes have any way of representing a threat
    model, very few have any way of even -expressing- the notion of
    a `false positive'.

    Note that this is distinct from the point I made (or tried
    to make) above: if you'll excuse the ornate diction,
    the lexical analysis/semantic content distinction is an epistemological
    point; the threat model contention is teleological. Somewhat more
    vaguely, the first distinction I made was largely about the structure
    of an IDS and the latter is about conceptual basis(-es) upon which
    value judgements are made by that NIDS.

    My point? Asking whether or not a NIDS reporting a signature match for a
    given packet is a `false positive' isn't even an insufficiently specified
    problem. It's like asking if OBP[3] is a better OS than SRM[4]. The
    question is poorly formed.

    - -spb

    - -----
    0 Or at least the -semantic- problem, here `semantic' meaning
    `having to do with semantics' and not the (more common in
    mailing list and USENET discussions) `nomenclature quibble'
    usage of the term.
    1 Gross conceptual error.
    2 I.e., a human operator.
    3 I (perhaps naively) expect most readers will still recognise
    this as Sun's Open Boot Prom.
    4 DEC's Systems Reference Manual, one of the boot consoles used
    on alphas and, earlier, VAXen.

    Version: GnuPG v1.2.1 (OpenBSD)

    -----END PGP SIGNATURE-----

    Network with over 10,000 of the brightest minds in information security
    at the largest, most highly-anticipated industry event of the year.
    Don't miss RSA Conference 2004! Choose from over 200 class sessions and
    see demos from more than 250 industry vendors. If your job touches
    security, you need to be here. Learn more or register at
    and use priority code SF4.

    Network with over 10,000 of the brightest minds in information security
    at the largest, most highly-anticipated industry event of the year.
    Don't miss RSA Conference 2004! Choose from over 200 class sessions and
    see demos from more than 250 industry vendors. If your job touches
    security, you need to be here. Learn more or register at
    and use priority code SF4.

  • Next message: Richard Bejtlich: "Re: Announcement: Alert Verification for Snort"

    Relevant Pages

    • RE: False Positives
      ... There isn't an IDS system that will not report "false positives" ... tools are not actually attacking but testing, and they report an attack, ... > IntruShield now offers unprecedented Intrusion IntelligenceTM ...
    • Re: Snot/state
      ... but not eliminate false positives by enabling this feature. ... > maintaining what the IDS considers state, ... maybe the ultimate IDS is only going to alert me to things that I ... they handle quite a few attacks - attacks that they are well aware of. ...
    • RE: Best Method(s) for signature verifcation.
      ... if the IDS is trying to be "smart" it may not listen on ports ... listening in order to get the IDS to see an attack. ... > Subject: Re: Best Methodfor signature verifcation. ... > false positives ...
    • Re: Is IDS/IPS worthless?
      ... >>firewall instead of in front of it should BOTH ... >>fill in the gap left by the false sense of security firewalls give (a ... >IDS technology and I certainly believe in the usefullness of IDS. ... that is confusing IDS and NIDS together. ...
    • RE: Truth about False Positives
      ... Subject: Truth about False Positives ... When using any kind of IDS wether it is host or network based first thing to ... defining false positives & false alarms, and what steps we are taking to ... algorithms into having the most comprehensive set of IDS attack algorithms. ...