Re: Random IDS Thoughts [WAS: Re: IDS thoughts]

From: Bill Royds (Bill_at_royds.net)
Date: 06/13/03

  • Next message: Mayank-Bhatnagar: "IDS and NMS" 
    To: "Steven Rudolph" <srudolph@iocenter.net>, <focus-ids@securityfocus.com>
Date: Thu, 12 Jun 2003 22:29:26 -0400

    Microsoft has a program called LogParser that takes IIS and other system
    logs (Event IIS error NCSA W3C etc.) and converts them to other formats or
    adds them to a SQL server/Access/ODBC data base. It has a builtin SQL engine
    to allow one to use a SQL syntax to select which logs to convert, including
    a number of functions
    It is free software but of course only works on Windows.

    ----- Original Message -----
    From: "Steven Rudolph" <srudolph@iocenter.net>
    To: <focus-ids@securityfocus.com>
    Sent: Tuesday, June 10, 2003 8:43 AM
    Subject: RE: Random IDS Thoughts [WAS: Re: IDS thoughts]

    Mike,
    Thank you for sharing this with everyone.
    You had mentioned that you have home grown your log collection system.
    If you are using any open source programs to do this, what have your
    choices been? I am attempting to build what sounds like a similar setup
    but on a much smaller scale - about 500 servers or so with sustained
    bandwidth in only the 10Mb range out to the net.

    I am still in the development/proof of concept stage and experimenting
    with different ideas at the moment. I would like to consolidate logs
    from syslog (using msyslog), Windows (syslogNT), and application logs.
    I am just starting the hunt for application log -> SQL database import
    utilities for both Apache, IIS and some others. Could you recommend any
    programs that are capable of doing this?

    Could you point me towards some papers or web sites that overview data
    mining techniques?
    Thanks,

    Steve Rudolph, CCSA, CCSE
    Internet Operations Center

    -----Original Message-----
    From: Mike Lyman [mailto:mlyman@west-point.org]
    Sent: Saturday, June 07, 2003 1:52 PM
    To: focus-ids@securityfocus.com
    Subject: RE: Random IDS Thoughts [WAS: Re: IDS thoughts]

    > Hint: data mining techniques, anyone ? There's a great book
    > by J. Mena on
    > the topic, which I warmly recommend.

    I don't think I've posted here before so to set this up, I've been
    running and building the IDS systems on a global network for about three
    to four years. 60,000+ employees and contingent staff, 300,000+ systems
    on the network and Internet egress and ingress in over two dozen
    locations around the world. Data overload is an understatement for what
    we face.

    The value of data mining on IDS data was first demonstrated to us by
    folks in our research group who had wanted to do a project on our IDS
    pilot data. They showed us stuff we'd have never seen even with today's
    consoles on the commercial IDS systems we use. Since that time we have
    more and more mining the data and twisting it this way and that. The
    single most common skill we put on job requirements is the ability to
    run SQL queries and that is a high priority on our training schedules.

    Through developing differenent views of all the data available to us and
    constant analysis, we've been able to create reliable alerts with few
    false positives from our commercial systems. With home grown log
    collection, we've been able to craft low noise, high signal alerting IDS
    systems from normal high noise event logging. All of it is finely tuned
    for our environment instead of generic enviroments that the IDS venders
    have to try to shoot for. It is no where near pefected yet but it is far
    more managable that what we used to have even though we now have
    considerably more data sources.

    If you are not looking into data mining techniques, you are missing a
    great way to use your data and reducing the data overload.

    Mike Lyman
    CISSP
    mlyman@west-point.org
    pgp keyid 0xD7BBADAD

    ------------------------------------------------------------------------
    -------
    INTRUSION PREVENTION: READY FOR PRIME TIME?

    IntruShield now offers unprecedented Intrusion IntelligenceTM
    capabilities
    - including intrusion identification, relevancy, direction, impact and
    analysis
    - enabling a path to prevention.

    Download the latest white paper "Intrusion Prevention: Myths,
    Challenges, and Requirements" at:
    http://www.securityfocus.com/IntruVert-focus-ids2
    ------------------------------------------------------------------------
    -------

    -------------------------------------------------------------------------------
    INTRUSION PREVENTION: READY FOR PRIME TIME?

    IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities
    - including intrusion identification, relevancy, direction, impact and analysis
    - enabling a path to prevention.

    Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at:
    http://www.securityfocus.com/IntruVert-focus-ids2
    -------------------------------------------------------------------------------

  • Next message: Mayank-Bhatnagar: "IDS and NMS"

    Relevant Pages

    • RE: Rather funny; looks like page defacement to me
      ... another security tool (VA, AV, firewall, etc.) that could have done the job ... I am not saying the IDS are always useless, but they are most useful as ... they denigrate Intrusion Prevention Systems and hail ...
      (Focus-IDS)
    • RE: Random IDS Thoughts [WAS: Re: IDS thoughts]
      ... Subject: Random IDS Thoughts ... today's consoles on the commercial IDS systems we use. ... INTRUSION PREVENTION: READY FOR PRIME TIME? ...
      (Focus-IDS)
    • Re: Random IDS Thoughts [WAS: Re: IDS thoughts]
      ... > running and building the IDS systems on a global network for about three ... > If you are not looking into data mining techniques, ... > INTRUSION PREVENTION: READY FOR PRIME TIME? ... > IntruShield now offers unprecedented Intrusion IntelligenceTM ...
      (Focus-IDS)
    • Re: IDS thoughts
      ... who most consider the mother of Anomaly Detection (because of her ... Subject: IDS thoughts ... > INTRUSION PREVENTION: READY FOR PRIME TIME? ...
      (Focus-IDS)
    • Re: Rather funny; looks like page defacement to me
      ... IDS can aleart you to an agresive scan form one ... security but I've seen to many times someone puts all thier trust in one ... INTRUSION PREVENTION: READY FOR PRIME TIME? ... Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at: ...
      (Focus-IDS)