Re: [fw-wiz] parsing logs ultra-fast inline

On 2/7/06, Marcus J. Ranum <mjr@xxxxxxxxx> wrote:

I think it's because a lot of webserver analysis tools are designed to
rip through the data and provide statistical summaries and sorted
hit-lists, whereas the security-oriented log processing tools are
aimed at audit functions. Since the security problem is less well-bounded
than "show me the top 50 pages on my site!" the designers of those
systems often reach for the biggest hammer in their toolbox and
stuff everything into a SQL database, which promptly falls over,
leading them to conclude "it can't be done."

Picking on me again already! Sheesh...

Okay, so I've gotten them to order some more ram and drive space for
my linux box. Going to start very small with one or two of our
internal PIXen...see how it goes. Still have no idea, really, how to
configure syslog-ng and write a perl script as described - but I'll
fumble through it.

Question: Better to do it inline or off-line (for starters anyway)? I
will turn it on for a day or so just to collect the first set of data
to begin writing the scripts with. Pretty sure syslog-ng will allow me
to create logs based on sources, so I figure it would require less
overhead to analyze individual files by type (and therefore similar
messages) like all of the PIXes, all of the ??Routers, AIX boxes,
etc.. I hate thinking about writing scripts for a month per device
type, but...

Second question: Hasn't anyone else ever written these scripts? You
would think they'd be pretty widely available - especially for things
like a PIX or 2600 or AIX. I mean, yes they're site specific but if
you know all of the errors/messages a PIX can provide (someone said
26k or so?) then the "meat" of a script could be generic enough...the
most common messages aren't likely to differ by much from site to your IPs/whatever in and run... or start to run...??
firewall-wizards mailing list