RE: [fw-wiz] parsing logs ultra-fast inline

marcus has been sufficiently saying what i do that i've not felt obliged to
participate in this thread, until finally:

-----Original Message-----
From: Marcus J. Ranum [mailto:mjr@xxxxxxxxx]
Sent: Wednesday, February 01, 2006 1:04 PM
To: firewall-wizards@xxxxxxxxxxxxxxxxxx
Subject: [fw-wiz] parsing logs ultra-fast inline

<<juicy stuff clipped for brevity>>

It comes back to what I was saying in my last few postings on this
- look at your data
- then decide what you want to collect from it
- then collect it
- count the stuff you throw away
- vector the rest to a human's attention

or take this list back one step further. **before** you look at your data,
think about what you'd like to know! as has been said repeated, it's hard to
build anything comparable to a human brain (of whatever generation) for
extracting useful data; on top of that, log analysis gets really hard when
you take a month's worth of data and say "tell me what i need to know from

but there's a big way to simplify, both in terms of technical complexity and
in terms of getting motivated to do it, *especially* if you're in the
position of just starting out.


so f'r instance, imagine i've landed in a new job at a company without a
centralized logging infrastructure. the network is the usual conglomeration
of file servers, mail, web stuff, firewalls, routers, remote access. and
databases, of course. and some custom code. i'd go MAD if i tried to build
the uber-logging facility all in one go.

but i have one or two types of "reports" (if you want to call them that)
that i've found particularly useful in the past, based on my experience as a
firewall and VPN admin. here, i want to start by building a logging system
that will give me daily reports on VPN access (who used it, and where'd they
come from). very targetted, very finite, very plausible.

when i've got that little bit working, i can add reports on administrative
activity on the VPN concentrators, and then the firewalls.

you see where i'm going. the people who run the boxes *know the kinds of
events they care about*. they may not have any idea about how to tell
whether that thing happened, but that's a different issue.

the actual point being that if you use what you know at the very beginning,
to target how you build things, you can make it a lot easier. i suppose you
run the risk of building something so specific to VPN logs that you can't
generalize it, but at least for me, i try to work with reasonable
architecture tools that let me branch out.

and even if it's completely specific to VPN logs, it has the great advantage
of getting that particular task done, rather than spending a lot of time
trying to start by building the perfect generic parser, and getting bogged
down, and never delivering.

cheers - tbird

firewall-wizards mailing list

Relevant Pages

  • Re: Router-to-router VPN
    ... Have made sucessful router to router vpn connection - great! ... For use to traveling laptops that vpn to the server. ... "firewall" is acting as the VPN Server and that the VPN Server is not behind ... terminating at the firewalls which is not the same thing. ...
  • RE: RDC and VPN
    ... We connect to VPN over WAN. ... When connecting through VPN, please ensure you can ping the target ... Reports sent via any other method will not be processed. ... The CAB file will contain the reports ...
  • Re: AD Consolidation Question
    ... Each forest represents a seperate location across the US and there are ... by firewalls and replication over firewalls, ... If you use a VPN and don't filter on those VPN interfaces the info on ...
  • Re: AD Consolidation Question
    ... BLOG --> ... Each forest represents a seperate location across the US and there are ... by firewalls and replication over firewalls, ... If you use a VPN and don't filter on those VPN interfaces the info on ...
  • RE: Firewalls on VPNs - Best Practice Advice
    ... | Subject: RE: Firewalls on VPNs - Best Practice Advice ... | Microsoft Online Partner Support ... | | I use the PC for connecting to various remote networks via VPN. ... | | connections is not recommended, ...