Re: Re: IDS Correlation

From: Hui Lee (huili@sei.xjtu.edu.cn)
Date: 03/28/02 

Date: Thu, 28 Mar 2002 12:44:54 +0800
From: Hui Lee <huili@sei.xjtu.edu.cn>
To: "prm@complexsys.net" <prm@complexsys.net>

>On Friday 22 March 2002 00:07, Matthew F. Caldwell wrote:
>
>> Simple Correlation
>> The ability to see all events in a normalized format side by side in a
>> single perspective. Example:
>
>This is, IMO, "data fusion" and not correlation. There seems to be quite
>a bit of marketing mis-use of various terms. This is NOT any criticism
>of Mr. Caldwell, it's just that I'm replying to his reply. Matt, I'm not
>jumping all over you for your post :-)
>
>I would throw out the following definitions as a strawman for discussion:
>
>Aggregation is the capability to display event information from multiple
>collection platforms in one location. Example: the Realsecure Management
>Console. This does not perform data reduction; there is still a 1:1
>relationship between events reported by the sensor array and events
>displayed at the analysis station.
>
>Data Fusion is the ability to display sensor events, collected from a
>heterogeneous collection array, sorted by _triggering sequence_.
>In other words, a given triggering sequence occurs on the network,
>and five different platforms record the traffic in five different ways. A
>data fusion system will provide the analyst with a single "event", and all
>the contributing events will be grouped into it. Matt's example, which
>I cleverly deleted, is an example of data fusion.
>
>Correlation is the ability to determine a pattern between atomic events
>(which are single-packet or single-rule-match alarms or events, whatever
>you want to call them). For example, an analyst can examine the
>atomic event stream of a Code Red II attack, recognize the pattern in
>it, and say, "that's Code Red II". This is manual correlation. An
>analysis system can provide correlation by recognizing a pattern,
>assigning it an identifier, and then looking for that pattern as a repeating
>event. For example, I designed a mathematical correlation system for
>Brinks Internet Security that can provide event correlation with 1000:1
>event reductions (assuming non-correlated events are uninteresting).
>That system uses a mathematical model for defining correlation. It
>was able to identify the pattern "1 ICMP echo request, 1 second
>delay, 1 ICMP echo request, 10 second delay, SSH connect attempt"

3x a lot for Mr Mover give a clear discrimination between
aggeration,data fusion,and correlation.As to correlation,
I want to employ Bayesian Network or goal-tree to do the similar
modelling of event stream,Do you think it is feasible
and how to acquire the structure and parameters,can you give me some exposition
about your mathematical model?
How to build it,and trained based on which kind of data?

>because that pattern occured more than once. The system was never
>actually programmed to look for that pattern.
>
>Note that true correlation systems are not rule-based systems, in that
>they're looking for patterns they *don't* recognize, rather than patterns
>they *do* recognize.
well,I think it is just a machine learning issue,as I know,all ML,espically
unsupervised ML,are not very successful,how do you acquire your ML model
and guarantee your models are effective?



>
>> Cross Organization Correlation
>> Managed Service Providers and Intelligence Service Providers can do this,
>> (or at least I hope so) you are trying to match similar events with a bit
>> of fuzziness to help customer come together as group in combating a threat.
>
>This is actually what the intelligence community calls I&W, or indications
>and warnings, activity. It's separate from, but related to, event
>correlation.
>
>> Time by Event Correlation
>> Statistics Table holds key information about Event counts and Time period
>> over hosts. Picks up slow scans and that type of thing if you have lots of
>> memory :>
>
>Again, there are mathematical processes, which we can convert into
>algorithms, that vastly increase the efficiency of this search. Those
>with a physics background will note that time is, well, time, in general,
>isn't what most people think it is. :-) If you change "time" from "the
>passage of a certain number of cycles in the CPU" to "the passage of
>a certain number, which may be dynamic, of some other 'thing'," then
>you can have "time" passing at different rates for different events, or
>even have multiple time sequences for the same events.

I can not understand the definition of time:)

>
>Sorry, I digress. It's just that asking a simple question like, "how does
>the system gauge time?" can lead to some astoundingly powerful
>analysis algorithms. (and that's just one pretty simple example....)
>
>> Rules or Pattern Correlation
>> This is a manual process of configuring rules to act as a higher-level
>> signature. IISUNICODE1 + IISUNICODE2 = NIMDA
>> Grouping Correlation
>> This type primarily called classes this would group all of your ids events
>> into a similar fashion. For example: ids.detect.ddos might contain many
>> DDOS signatures.
>
>Again, this is just a high level rule engine. See the previous thread
>between MJR and myself with the subject "Statistical anomaly detection".
>
>The bottom line here is that currently available correlation techniques
>are the Lincoln Logs of the construction industry. There are, in research
>labs here and there, such as UC Davis, groups of people working on
>more advanced techniques, but there is little to no commercial demand
>for such products. Example: there are a number of organizations out there,
>such as EDS's Information Assurance program, AFIWC (the Air Force
>Information Warfare Center in Plano, TX), RipTech, and Brinks Internet
>Security, that provide advanced I&W analysis capabilities, but the
>customers who are actually willing to PAY more for that capability are
>few and far between. IOW, it's not a perceived benefit to the consumer.
>If there isn't profit in it, commercial providers won't be in a hurry to field
>it.
>
>Companies and organizations that want a true advanced IDS/event
>correlation system are going to have to pony up the research dollars to
>build it themselves. And they won't find the talent or technique, for the
>most part, in the IT community. They'll find it in physics, signal processing
>electrical engineering, computer science, sociology, cognitive
>psychology, and evolutionary computing, to name a few.
>
>Sorry again. I do tend to go on and on about these things.... :-)
>
>Regards,
>Phil
>
>"Reality, like truth and beauty, is in the eye of the beholder."
> - John L. Casti

Quantcast