AW: Views and Correlation in Intrusion Detection

From: Liesen, Detmar (LDS) (Detmar.Liesen_at_lds.nrw.de)
Date: 07/01/03

  • Next message: John Levine: "Use of Honeynets to Secure Large Enterprise Networks"
    Date: Tue, 1 Jul 2003 07:32:30 +0200
    To: "Richard Ginski" <rginski@co.pinellas.fl.us>, <focus-ids@securityfocus.com>
    
    

    FYI: There is a project on Sourceforge that aims at devoloping an
    event-vulnerability correlation tool:
    www.sourceforge.net/projects/threatman

    The licence will be GPL 2.0.

    Unfortunately we lack of competent developers who can afford investing some
    amount of time into this project.
    Until now we have mainly done development-planning and basic-design.

    We want to build an open architecture for threat-management that uses IDMEF as
    normalized message-format and IDXP over TLS (with Roadrunner) as communication
    protocol. The architecture will be multi-tiered and provide plug-ins for a
    variety of IDS-products.

    Our first implementation will be a snort plugin and a nessus plugin, so that
    snort events and nessus reports can be correlated.

    If you have the time and skills you're invited to participate...

    Feel free to browse the devel-docs in our CVS repository.

    Greetings,
    Detmar

    -----Ursprungliche Nachricht-----
    Von: Richard Ginski [mailto:rginski@co.pinellas.fl.us]
    Gesendet: Donnerstag, 26. Juni 2003 21:20
    An: focus-ids@securityfocus.com
    Betreff: RE: Views and Correlation in Intrusion Detection

    Warning...possible stupid questions below:

    Doesn't a major component of such a thing already exist with Intrusion
    Detection Message Exchange?

    http://www.ietf.org/internet-drafts/draft-ietf-idwg-requirements-10.txt

    If so, is it just a matter of vendors wanting to "play together" and
    implement it in their products? I'm curious if this is the real
    stumbling block. It seems that correlation has been discussed for a
    while (years). From what I've experienced, technology doesn't take this
    long to develop unless "people" don't want to.

    >>> "Sekurity Wizard" <s.wizard@boundariez.com> 6/25/2003 11:03:04 PM
    >>>
    David,
            Your are all absolutely correct - correlation is the gold
    medal...right now everyone in the industry is praying for bronze at
    best. The one glimmer of hope I see are products out there, and I
    don't
    remember the company name right now, that aggregate hundreds of
    gigabits
    of logs per hour and try to make sense of it all. The question them
    becomes one of scalability...assuming we take for granted someone CAN
    write an engine that processes this sort of data in a sane manner.
            Scalability, in the form of the type of environment I work at
    is
    insanely large. We have umpteen numbers of DS3's, countless T1's and
    thousands of pipes to and from segments we aren't even *aware
    of*...not
    to count the couple of hundred (close to 1,000) firewalls that are out
    there. Now, let's say we put a couple of these boxes (~50Mbit/sec
    each)
    to the test in my environment. There STILL NEEDS TO BE A CENTRAL
    PROCESSOR...otherwise, we're left with the distributed view - which we
    don't want, right? Is it realistic to think there is such a scalable
    system that can process hundreds of gigabits of data per second,
    aggregate it all, normalize it, and correlate too? I dare say not at
    this point...unless we come up with some sort of standard, "XML for
    security devices" that makes the processing and data crunching
    easier....but the problem there is I don't see Checkpoing, Cisco,
    Enterasys, and ISS (and others) getting together on this any time
    soon....

            So scalability is our main opponent as I see it...because at
    the
    end of the day - the only attack that counts is the 1 in 100,000,000
    that sent that single UDP packet that triggered a shutdown of the
    entire
    network due to SQL Server port floods...right?

    Sleep well... :)

    -----Original Message-----
    From: DAVID MARKLE [mailto:davidmarkle@comcast.net]
    Sent: Tuesday, June 17, 2003 1:49 PM
    To: Blake Matheny
    Cc: focus-ids@securityfocus.com; davidmarkle@comcast.net
    Subject: Re: Views and Correlation in Intrusion Detection

    Blake, I agree with your sentiments regarding correlation and have more

    to add.

    The point of correlation is the value it adds to mostly autonomous,
    unreviewed, and meaningless data. (The folks that disagree with this
    line must have economically independent budgets with staffing
    consisting of superstar (I applaud you)). Who reviews the firewall
    logs? I don't. We have over 500 global firewalls. The point here is

    (as you stated) AUTOMATION. But it does not stop there. That data has

    to be normalized and applied towards something. The correlation piece

    adds that middleware "something". An IDS alert is ONLY relevant if the

    firewall permits the traffic through. To further the comment, and
    attack signature tripped for (known attack) xyz, is ONLY relevant when

    the attacked host is vulnerable to xyz. This is the ultimate job of
    correlation. If the above surrounding conditions are true, the
    severity of the attack becomes increased to critical, otherwise it is
    informational only. There are also netops statistics that should be
    considered security related (and monitored). Baseline your bandwidth,

    averaged over 12 months. Normal increases in business offerings are
    roughly 5 percent per month. Since there was no change control this
    past weekend (to relate), why did you see a spike in bandwidth by 17
    percent ???? Why is tcp 2148 increasing on your global perimeter over

    the past 3 days? These are relevant questions. Without the collection

    and aggregation of the appropriate data, we run the operations in the
    dark.

    With regards to the state of correlation, I still think its an infancy

    issue. Historically, I believe that the industry (tech folks) has been

    extremely focused on growth development and deployment of the
    technology (firewalls, IDS-(H/N), etc.). Firewalls have been around
    for awhile and have matured to a point of plateau (mostly). IDS is now

    in "the growth phase" (with heuristic, anomaly, signature, blah, blah,

    blah), and all that hype. I really think that the industry had
    recently realizes that we are now overwhelmed with too much data. Now
    everyone is scrambling to catch up .....

    David Markle
    davidmarkle@comcast.net
    davidmarkle@elephantfoot.org

    ----- Original Message -----
    From: Blake Matheny <bmatheny@mkfifo.net>
    Date: Tuesday, June 17, 2003 1:32 pm
    Subject: Views and Correlation in Intrusion Detection

    > Two areas that I have recently been doing research in, are views
    > and their
    > connection to correlation techniques. In terms of systems, given
    > some event,
    > the information we get about the occurrence of such an event comes
    > to us in
    > the form of either a primary or a secondary view. Information
    > about secondary
    > views typically come to us from applications such as firewalls and
    > ID systems.
    > Primary information usually is received from the application
    actually
    > processing this data for use. For instance, an ID sensor may
    > produce an alert
    > about some traffic. However, this is a secondary view of the event
    > and needs
    > to be correlated with other, relevant information. So of course
    > firewall logs
    > might be checked, to see if traffic actually passed that
    > corresponds to the
    > event in question. This is also a secondary view, so a third place
    > is checked,
    > the applications logs.
    > There are really several issues here. First of all, a tremendous
    > amount of
    > time is being spent, trying to correlate all the relevant
    > information. This is
    > something that _can_ be automated. Second, the applications logs
    > may not be
    > trustworthy. Third, and to me, most importantly, is the fact that
    > this is such
    > a 'basic' thing that people using ID systems have to do, and there
    > is no piece
    > of software yet that does this.
    > So something we have been working on, is a system to deal with
    > this basic
    > type of scenario. This will entail data transformations into an
    > intermediarylanguage, an event description language, offline state
    > analysis and several
    > other components (there is more information at
    > http://www.nongnu.org/babe/).If you spend some time thinking about
    > everything involved to do this in a
    > scalable fashion, it's an enormous task (I said basic, not
    > simple). What I am
    > finding frustrating, is that much of the base research has not yet
    > even been
    > done. Much of the research that has been done, is either too
    > primitive or too
    > impractical to be implemented. Is this due to the infancy and
    > immaturity of
    > the field, do people not see this as being feasible and therefor
    > aren'tspending the research time, or is this simply too far down
    > the line? In any
    > case, feedback welcome. Thanks.
    >
    > Cheers,
    >
    > -Blake
    >
    > --
    > Blake Matheny "... one of the main causes of the fall of
    the
    > bmatheny@mkfifo.net Roman Empire was that, lacking zero, they
    had
    > http://www.mkfifo.net no way to indicate successful termination
    of
    > http://ovmj.org/GNUnet/ their C programs." --Robert Firth
    >
    > -------------------------------------------------------------------
    > ------------
    > Attend the Black Hat Briefings & Training, July 28 - 31 in Las
    > Vegas, the
    > world's premier technical IT security event! 10 tracks, 15
    > training sessions,
    > 1,800 delegates from 30 nations including all of the top experts,
    > from CSO's to
    > "underground" security specialists. See for yourself what the
    > buzz is about!
    > Early-bird registration ends July 3. This event will sell out.
    > www.blackhat.com---------------------------------------------------
    > ----------------------------
    >
    >

    ------------------------------------------------------------------------
    -------
    Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas,
    the
    world's premier technical IT security event! 10 tracks, 15 training
    sessions,
    1,800 delegates from 30 nations including all of the top experts, from
    CSO's to
    "underground" security specialists. See for yourself what the buzz is
    about!
    Early-bird registration ends July 3. This event will sell out.
    www.blackhat.com
    ------------------------------------------------------------------------
    -------

    -------------------------------------------------------------------------------
    Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas,
    the
    world's premier technical IT security event! 10 tracks, 15 training
    sessions,
    1,800 delegates from 30 nations including all of the top experts, from
    CSO's to
    "underground" security specialists. See for yourself what the buzz is
    about!
    Early-bird registration ends July 3. This event will sell out.
    www.blackhat.com
    -------------------------------------------------------------------------------

    -------------------------------------------------------------------------------
    Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas, the
    world's premier technical IT security event! 10 tracks, 15 training sessions,
    1,800 delegates from 30 nations including all of the top experts, from CSO's to
    "underground" security specialists. See for yourself what the buzz is about!
    Early-bird registration ends July 3. This event will sell out. www.blackhat.com
    -------------------------------------------------------------------------------

    -------------------------------------------------------------------------------
    Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas, the
    world's premier technical IT security event! 10 tracks, 15 training sessions,
    1,800 delegates from 30 nations including all of the top experts, from CSO's to
    "underground" security specialists. See for yourself what the buzz is about!
    Early-bird registration ends July 3. This event will sell out. www.blackhat.com
    -------------------------------------------------------------------------------


  • Next message: John Levine: "Use of Honeynets to Secure Large Enterprise Networks"