Re: [fw-wiz] Handling large log files



I have a central log server set up in our environment that would
receive around 200-300 MB of messages per day from various devices
(switches, routers, firewalls, etc).  With this volume, logcheck was
able to effectively parse the files and send out a nice email.  Now,
however, the volume has increased to around 3-5 GB per day and will
continue growing as we add more systems.  Unfortunately, the old
logcheck solution now spends hours trying to parse the logs, and even
if it finishes, it will generate an email that is too big to send.

Hi Nate,

I will offer a few general suggestions. You should be able to reclaim some
capacity from the system with review of the overall logging architecture and
your rule/reporting configuration. Harness the project's mailing list, and try
to profile your system in the hope of identifying easily addressed bottlenecks.

- log volume increase by an order of magnitude usually means that the
complexity of the environment quadruples. At this point, a 10% increase
in the size of your environment adds an equivalent of the original log flow.
I assume that, with adding more machines, the environment is getting more
standardized, but might have to look for a bigger tool.

- unless you have already done so, I would try to optimize the
ruleset. Make sure
that the logs go through as few regular expressions as posisble. With GB/day
of text, the cost of the extra evaluations compounds. Following the same logic,
investigate potential rewriting the most used, or the most expensive
rules. Try to
squeeze as much capacity from your install as possible.

- profile the machines, make sure that disk/network IO keeps up, that CPUs are
not running at 100% at all times, etc. This will let you identify
bottlenecks, and
further extend the live of your current system.

- scrap the existing reports. Write down the list of requirements, and
the nice-to-haves,
the scope and the needed level of details, and write new reports (you
should be able
to reuse most of the original work).

- see if the architecture can be improved. Can you use multiple log
servers? Is there
a logical way of segmenting the log traffic - OS to box 1, db
transactions to box 2, etc.?
Post to the project's mailing list, there should be people who use it
for larger installations,
and willing/able to provide specific suggestions.

- commercial tools should be able to keep up with 2gb/day without much
effort, but
every one will take considerable time to set up and tune. The vendors
will claim that it's
2 day setup and a week of rule setup, etc, but I would consider
planning for a quarter long
mid-intensity project. The end result should be useful dashboards and
reports that make
sense. I would set aside at least $20k, but that will be very
dependent on your environment.
Some products have reporting/integration plugins costing that much.

My team logs 2-3gb through Splunk, with no performance issues of any
kind (nice box
8cores/8gb ram). With a bit of careful planning, I expect to put quite
a bit more through it
in the near future.

--
Marcin Antkiewicz
_______________________________________________
firewall-wizards mailing list
firewall-wizards@xxxxxxxxxxxxxxxxxxxxx
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards



Relevant Pages

  • Re: .
    ... menus, twenty data entry screens, and fifteen reports. ... FileAid in an MVS, TSO/ISPF environment, with Windows 95 on site. ... and scenarios, testing and documenting. ... As a Senior Systems Analyst, I was responsible for: designing new interfaces for loading of data sent by the clients; ...
    (rec.video)
  • Re: Getting users domain name
    ... >> system - basic Windows XP english vs Windows XP german with something ... >> When they run the application, it reports their domain as their computer ... >> I checked their system environment variables (manually, ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Getting users domain name
    ... >> system - basic Windows XP english vs Windows XP german with something ... >> When they run the application, it reports their domain as their computer ... >> I checked their system environment variables (manually, ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Getting users domain name
    ... >> system - basic Windows XP english vs Windows XP german with something ... >> When they run the application, it reports their domain as their computer ... >> I checked their system environment variables (manually, ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Log Files
    ... > logcheck seems to be what i need. ... probably due to me not rotating my log files eh? ... severity it reports it. ... Remove all numbers, then remove invalid, email, no, and spam to reply. ...
    (comp.unix.bsd.freebsd.misc)