Re: Random IDS Thoughts [WAS: Re: IDS thoughts]

From: SecurIT Informatique Inc. (securit_at_iquebec.com)
Date: 05/30/03 

Date: Fri, 30 May 2003 15:25:28 -0400
To: focus-ids@securityfocus.com

I know I'm a bit late in this thread, but I was busy with preparing the
release of LogIDS 1.0 and accompanying tools. I'd like to comment the
first point from Greg's last message: (my comments are after the message quote)

At 06:54 PM 29/05/2003, Greg Shipley wrote:

>Interesting thread. I started to quote bits and pieces, and then decided
>to say "screw it" and just make this a much cleaner post. (Albeit, I
>suspect it's going to be a much *longer* post, but...) So here goes...
>
>A few observations, for whatever it's worth:
>
>----------
>
>1. Commodotization of the IDS space, in general: I was in the airport
>coming back from Networld Interop (Vegas) and one of my fellow attendees
>turned to me and said "Ya know, all these IDS vendors sound the same to
>me. I can't tell the difference anymore." I thought this was an
>interesting comment, and while I certainly don't agree that all IDS
>vendors ARE the same (far from it, in fact), I doubt I would have heard
>that comment even 12 months ago. This is a sign, IMHO.
>
>I think there *is* some amount of "convergence" in the space, and I'll
>suggest that there is a lot more commonality between products these days
>than there has been in the past. For example, recall the "packet
>grepping" vs. "protocol inspection" debate of a year ago; almost all NIDS
>products support both models now.
>
>But here's a thought for some to chew on: IMHO, there are different
>components to an IDS "solution" and I think the focus is starting to
>shift. I think we, as an industry (myself included), have been very
>sensor-centric in our views of NIDS...and I suspect that this is starting
>to change. When I talk to operators of enterprise NIDS deployments
>they're chewing my ear off about data management, and more specifically,
>data overload. The sensor has to do its job, yes, but the big hurdle many
>are facing is just digesting the amount of info coming at them.
>False-positives or not, let's face it folks, if you have dozens of
>devices, are monitoring busy networks, and aggregating your events
>(firewall, NIDS, etc.), there's a good chance you're going to be choking
>on a serious amount of data. So I think DATA MANAGEMENT is going to
>become (if it isn't already), a much bigger issue. And I'm not just
>talking log aggregation - I'm talking user interface, correlation, useful
>tools, etc.
>
>Will keeping up with new vulnerabilities, creating better engines,
>improving detection accuracy, supporting higher speeds, etc., still be
>challenging? Absolutely. But the sensor-based tech is just ONE
>component, and I think features like device management, data management,
>front-end interfaces (and tools to reduce analyst man-hours) are rapidly
>growing in importance, and are going to be some of the bigger hurdles
>moving forward.
>
>All this to say that perhaps the NIDS *sensor* technology is moving
>towards commoditization, but I think the overall solution has a far way to
>go...

The fact that most IDS products out there now look the same is based on the
fact that most companies out there (or the people running them, to be more
precise) know more about making money than designing new
technologies. Someone thinks about an IDs concept, then make a proof of
concept program, eventually improves it while in the meantime the program
gets community approval (ie, it gets used). The concept is further
studied, and eventually becomes better defined, or will offspring other
concepts, generating new tools also for these. But mostly, this is the
work of idividuals, not companies. What companies will do, when a
technology or concept is well enough mastered by the community, will start
producing commercial software applying these very same concepts in a
competitive market. They can have a few tricks of their own, but usually
they won't develop the whole concept by themselves. Such a concept would
be NIDS, which makes all NIDS work more or less around the same principles,
which makes all the products look a bit the same. Another concept would be
HIDS, and there you have another series of products based on this
particular idea. Of course, some companies will actually mix some of these
together, but the basic ideas behind the scheme are always the
same. Antivirus or firewalls could be other examples of this.

Another IDS trend that gets talked about every so often but is not much
seen in the wild is statistical-based IDS, ot anomaly-based IDS (what would
be the acronym? SIDS or AIDS?). It is also said that this could be beaten
by flooding a network with "anomalous" traffic so it eventually gets
considered as normal traffic and thus be circumvented in some way. What
will end up with this trend is that once the idea is workable in a prod
environment, companies will start making offering of software that will
apply this very specific idea.

Now, that leads to the data management problem. I've been working on log
collecting for quite some time now, and have often be faced with this
question: what about the volume of the logs?, implying that the staff
wouldn't have the time to parse through it. So I have been thinking about
ways to improve this and maybe even fix this problem. First of all, what
struck me most when I got confronted to this question, is that instead of
being noticed in real-time of events logged by antivirus and personal
firewalls (this was the context of the discussion, during some job
interviews), my interlocutors often found preferable the status-quo, which
was not being alerted at all. Now, I understand that analysing logs take
time, but what makes it the most time consuming task is the fact that we
often have to analyse it in large chunks at a time, covering a large
timeframe. Being notified of events as they occur takes less time, as you
only have to deal with the data presented at this time. Not knowing about
this data, or looking at it a long time after it occured, actually takes
the most time, as you have to understand what happened, and to what scale,
and start fixing things up to a point that you probably could have limited
damage if being notified earlier, and saved time at fixing less broken
things. My 2 cents about all this.

So, what came out of all of this? Well, as you can see, I tend to agree
with Greg's statement that a large volume of data submitted by various
devices on a network can take quite an amount of time, and is a problem in
itself. I think this problem has more than one cause, and here are what I
think makes log management a problem nowadays:

1) Lousy interface design : Most IDS products or log analyzer products I've
been exposed to, either by direct usage or by screenshots on the web, bare
a variation of the same graphical interface: a database of records
displaying the various fields contained in the log lines. Some of them let
you click on it to get a further explanation of the event, but basically,
this is not much different than looking at your logs in a text editor or in
Excel, except maybe for a few gimmicks like statistical charts of traffic
usage and such. These tools can be useful, but they don't quite make it
easier to analyse and understand the content of log lines.

2) Things work for themselves only : What I mean here is that security can
only be achived by applying several levels of security measures all accross
our network. The more measures or devices you add, on the more machines
you install them, the more logs it makes you to manage, but all
separately. ie: firewall logs in one file, antivirus logs in another
files, IDS logs in yet another files, etc... I think that "event
correlation" tries to fill that gap, but so far I don't think they quite do
it. I'll take the liberty to quote Marcus Ranum here from his speech at
Seguridad en Computo 2003 (Mexico City), where he said that event
correlation engines are practically nothing more than a software than
instead of displaying 60 000 times the same king of event logged, will give
one event saying that this have occured 60 000 times. Not much more of an
upgrade in the situation. Besides, for event correlation to mean anything,
IMHO, it should actually try to correlate events from the various logs
gathered from all our devices, instead.

3) No automatic analysis available : Most of the "log analyser" software
I've looked at in the last 2 years or so (I know I could have missed some,
but I looked pretty hard) are flawed in one way or another : either it
works for a single software only (leaving you with problem 2), either
"analyzing" means that it can break down the various field and display it
to you on a GUI (which leaves you with problem 1), either "analyzing" means
that it produces nice charts for upper management (which doesn't help at
all). Some will actually let you specify rules so you can have a first
automatic sift through the logs before you look at it, but they also suffer
from problem 2 and 1 too, so they can be of help, but this will not be THE
single interface to look at all your logged events.

I've been told of Guarded.Net neuSecure product, and it looks impressive,
but also expensive. It seems to be automatically compatible with a
smorgasbord of application logs, and is backed with a powerful database
engine, but I have not been able to find a single screenshot, so I don't
know if this product is also affected by problem 1.

So thinking about all that, I thought of designing a log-based IDS, or LIDS
for acronym fans. I'm not sure if this concept has been defined before,
but I'm sure the topic must have appeared in a conversation or two. I
haven't seen it often described as such though, and would like to do it
here. LIDS is actually an intrusion detection engine based on real-time
analysis of logs gathered from all across the network and from various
software sources, presented in one single interface. It is not like a
traditional IDS system, in the fact that it is not trying to find
incongruities with some aspect of the IT architecture; there is not point
in re-inventing the wheel, so there's no need to make it behave like a NIDS
or a HIDS when there are already these kind of tools around. Instead, it
will give you one interface to analyse data from both sources, but also
from antivirus, firewalls, personnal firewalls, keyloggers(ie ComLog),
etc... When you can see in one interface what are the various devices that
are reporting during an incident, and what machines are reporting among
your network, as the incident occur, this greatly improves the conditions
in which the person in charge of the network is in order to figure out what
is happening. If prior to being displayed, the data can be analysed to
discard informative-only logs or logs reporting non-suspicious or
innofensive events, then the better as this will help to reduce the amount
of data submitted to the administrator's attention.

Now, about the interface. This is maybe the most problematic aspect of
this whole issue. I tried to go beyond the usual interface of "displaying
log lines after log lines" usually seen in most log analysis
softwares. Also, when I think about "event correlation", to me it means
that you shoud be able to easily "correlate" events between machines, so
you can see what machines are involved in an incident, and how they are
involved between each other (ie, one machine is the launchpad, the other is
the victim, for example). I thought about this when thinking about the
interface, and I opted to make the interface to be a multi-windowed network
map, where each node on the network (either a host or a subnet) has its own
log monitoring window. The network nodes also have a small icon associated
with it, that can change in order to represent the activity depicted in the
logs it is analysing. It can also emit sounds for warnings and
alerts. The final result of this work gave birth to LogIDS 1.0, that I
released at the beginning of this week. You can see a screenshot of LogIDS
1.0 Free (Open Source) at
http://iquebec.ifrance.com/securit/image/figure1.gif and a screenshot of
LogIDS 1.0 Pro at http://iquebec.ifrance.com/securit/image/figure10.gif
(the pro version contains some automatic analysis and rules, and display
separate windows for command prompt sessions captured by ComLog).

This is the first version of this tool, I hope people will find it usefull
and feed me with their feedback and suggestions at improving the design and
the code. In particular, I'd like to hear what people think of the
interface. Do you think it actually improves at understanding the events
at hand? Or do you think that the reduced window size is too
small? Knowing that bigger windows mean less object space, what is better:
bigger windows for less objects, or the current object size currently fits
the needs to portray a real-life network? Well, that kind of thing.

So, as I said, LogIDS 1.0 is available both in Open Source and commercial
versions, so that security enthousiasts can take a look at my work and let
the flow of ideas go freely, while offering a solution for companies that
would like the enhanced features of the Pro version. But apart from the
pop-up windows for ComLog, it is possible to achieve more or less the same
results with both versions. I have tried to make it as flexible and as
customizable as possible, so that it can fits most people's needs. I also
tried to make it as vendir-independant as possible, but it is especially
fit for use with my other tools (ComLog, a command prompt keylogger, and
LogAgent 4.0, a log monitoring and centralising software, that you can use
to forward logs from any application (except IIS) to wherever you
want). You can specify the field descriptions of every log file you want
to monitor, and then you can apply rules using these fields. Rules can be
to display an icon representing the action depicted in the logs, emit
sounds, display the information in the appropriate window in the GUI (you
can also select which fields you want displayed), drop the record to the
console (used to display "garbage" information) or dropped altogether (gets
ignored). Note that the way the software works, if a record gets dropped
or rejected, if other rules apply to it, they will still be activated.

You can download LogIDS 1.0 (both versions), LogAgent 4.0 (both versions)
and ComLog 1.05 (Free) from my website
http://securit.iquebec.com/. Comments and feedback are welcome. All these
tools are in Perl, and the doc is in the zip files.

I'd also appreciate if people could look at the code and suggest me ways to
improve it, either in terms of performance or in terms of "good
programming". I tried to take good care of riding all the bugs, but I may
not be totally exempt of a BO of some sort. Although I don't think that
this program can be the main door for an intrusion, and that if your
systems are getting hacked while being monitored by LogIDS, you should have
been notified well before the console itself gets attacked, IMHO. Still,
I'm sure that this is a great idea, but that it is called to evolve, in
many ways. I do not think that is somethng I will be able to do all by
myself, and I hope LogIDS raises enough interest in the community to make
it as much a must-have as nmap, snort or netcat in the long run.

Adam Richard, aka Floydman
SécurIT Informatique Inc.
securit@iquebec.com
http://securit.iquebec.com/

-------------------------------------------------------------------------------
INTRUSION PREVENTION: READY FOR PRIME TIME?

IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities
- including intrusion identification, relevancy, direction, impact and analysis
- enabling a path to prevention.

Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at:
http://www.securityfocus.com/IntruVert-focus-ids2
-------------------------------------------------------------------------------