Re: [fw-wiz] Handling large log files



I have been using rsyslog (as opposed to syslog-ng) and found it to be
quite useful. It is under very active development and the main
developer is REALLY into logs.

sai

On Thu, May 7, 2009 at 12:56 AM, <hugh.fraser@xxxxxxxxxxxxxxxxx> wrote:
Like others have mentioned in previous replies, we've used syslog-ng and
Splunk to manage firewall and switch event logs. But sometimes we've
wanted to detect behaviour or anomalies that can't be done easily with
the tools. For these, I've used SEC (Simple Event Correlation), and perl
script from:

http://kodu.neti.ee/~risto/sec/

During the replacement of our campus network when lots of inter-switch
dependency issues arose, we used it to alert us to switches reporting an
error that hadn't had any problems for the past 5 days, usually
indicating something had happened externally to affect it, or to events
that were new in the past 5 days. We also used it to identify things
like links bouncing (down/up/down within a certain period of time). The
output of SEC was fed back in to syslog-ng as and represented in Splunk
as "synthetic" events, for which we had special notification and
reporting.

The goal of the process was to do exception reporting, allowing us to
collect all the events but only be notified when certain criteria
occurred.



-----Original Message-----
From: firewall-wizards-bounces@xxxxxxxxxxxxxxxxxxxxxxx
[mailto:firewall-wizards-bounces@xxxxxxxxxxxxxxxxxxxxxxx] On Behalf Of
Nate Hausrath
Sent: Tuesday, May 05, 2009 6:41 PM
To: firewall-wizards@xxxxxxxxxxxxxxxxxxxxxxx
Subject: [fw-wiz] Handling large log files

Hello everyone,

I have a central log server set up in our environment that would receive
around 200-300 MB of messages per day from various devices (switches,
routers, firewalls, etc).  With this volume, logcheck was able to
effectively parse the files and send out a nice email.  Now, however,
the volume has increased to around 3-5 GB per day and will continue
growing as we add more systems.  Unfortunately, the old logcheck
solution now spends hours trying to parse the logs, and even if it
finishes, it will generate an email that is too big to send.

I'm somewhat new to log management, and I've done quite a bit of
googling for solutions.  However, my problem is that I just don't have
enough experience to know what I need.  Should I try to work with
logcheck/logsentry in hopes that I can improve its efficiency more?
Should I use filters on syslog-ng to cut out some of the messages I
don't want to see as they reach the box?

I have also thought that it would be useful to cut out all the duplicate
messages and just simply report on the number of times per day I see
each message.  After this, it seems likely that logcheck would be able
to effectively parse through the remaining logs and report the items
that I need to see (as well as new messages that could be interesting).

Are there other solutions that would be better suited to log volumes
like this?  Should I look at commercial products?

Any comments/criticisms/suggestions would be greatly appreciated!
Please let me know if I need to provide more information.  Again, my
lack of experience in this area causes me hesitant to make a solid
decision without asking for some guidance first.  I don't want to spend
a lot of time going in one direction, only to find that I was completely
wrong.

Thanks!
Nate
_______________________________________________
firewall-wizards mailing list
firewall-wizards@xxxxxxxxxxxxxxxxxxxxx
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards


_______________________________________________
firewall-wizards mailing list
firewall-wizards@xxxxxxxxxxxxxxxxxxxxx
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards

_______________________________________________
firewall-wizards mailing list
firewall-wizards@xxxxxxxxxxxxxxxxxxxxx
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards



Relevant Pages

  • Re: [fw-wiz] Handling large log files
    ... Splunk to manage firewall and switch event logs. ... we used it to alert us to switches reporting an ...  With this volume, logcheck was able to ... effectively parse the files and send out a nice email. ...
    (Firewall-Wizards)
  • Re: Logging
    ... Thank you I actually have syslog-ng installed and I am working on install Splunk, thanks for the feed back everyone. ... Subject: Logging ... splunk is just ok for viewing logs (not very nice to ...
    (Security-Basics)
  • Trying to colour syslog-ng logs to ttyv7 but wont work after a reboot
    ... I have a 5.4-STABLE server that I've reconfigured to use syslog-ng ... It collects logs from all our servers and sorts ... I set syslog-ng to log all remote logs to this destination, ... for some reason way beyond me, it *will not work* after a reboot. ...
    (freebsd-questions)
  • Re: Scheduled Server scan does not log events - Trend Micro WFBS 5.1
    ... reporting and logging facilities. ... Query-Exchange Server-Scan event logs. ... can set the cpu utilization to high, ... We have recently discovered that our Sunday morning Scheduled Server Scan ...
    (microsoft.public.windows.server.sbs)
  • Re: [fw-wiz] syslog and network management
    ... syslogd we were able to handle an order of magnatude more logs ... How was syslog-ng implemented? ... Which debian ... we noticed a LOT of missing logs, when we changed to the default ...
    (Firewall-Wizards)