Re: [fw-wiz] RE: IDS (was: FW appliance comparison)

On 1/25/06, Marcus J. Ranum <mjr@xxxxxxxxx> wrote:
> Brian Loe wrote:
> >Where I work, I'm not sure how we could do it. We're a transactions
> >company, and do thousands and thousands (and more at times) a second.
>
> Would you like to think about that for maybe a second?? Logging
> an event is, what, thousands of times less CPU and I/O intensive
> than executing a transaction?? So how can you say that you're
> not sure how to do something that's _easier_ than what you are
> already doing??

To run the transactions they have a VERY large mainframe. To collect
logging I'm lucky to have gotten (since they got it for free
themselves) a pseries running linux. Slight difference.

> Sure, it's not something you'd want to handle with lightweight
> tools or slow interpreted programming languages, but you are
> not talking about spine-crushing data rates.

Without the sarcasm I fully understand what you are saying. But you
don't have to convince me, I have to convince my management - which
doesn't see a problem. They're running debug on every device they own
right now, they're just not logging it, tracking it, analyzing it..or
anything else with it - until there's a problem. You're stating that
they have to spend money - at least for disk space. I'd be laughed
at...unless IBM or Cisco can do it with a "device".


> Syslog definitely has problems with high rates of input. See:
> http://lists.jammed.com/loganalysis/2002/01/0054.html
> but it's mostly due to UDP output queue overruns.


I'll take a look...

>
> It's not a hardware problem... But - wait - you said "database"?
> Please tell me you weren't trying to stick that much data into
> a SQL database with indexes on your tables and an interpreted
> query/optimizer engine on top of all that? If so, I'm not surprised
> it didn't work -- but that's not a "logging is hard" problem that is
> a "using a relational database for a write-heavy application is
> the wrong tool" problem.

I didn't realize what I was getting into, firstly. Secondly, what good
does the data do if you can't "do" anything with it? Without a system
to at least *help* you analyze it you're simply swimming in quicksand,
flailing in fact.


>
> > With that much data, and 98% of it being useless, you kind have
> >to ask yourself, "what's the point?"
>
> I don't ask myself that. Because I don't agree that 98% of it is
> useless. It's probably closer to 99.99999% of it is useless.
> Except for the one or two lines that you might someday
> really, really need.

Some day, somewhere, something like that might happen. Building a
business case for purchases on that line of reasoning doesn't work so
good though. If you know of a better way of doing this that doesn't
cost money, I'm all ears and management, well, they won't care either
way.

>
> That's the problem, then. You're assuming that your IDS is going
> to know how to detect some site-specific hack that only works
> against you. That's what the logging is for. It's for figuring out
> what happened after it's too late.

That's a good point - and one I hadn't though of. As for IDS, I
personally think its a mostly useless tool - especially the way they
have it implemented here.


> Sometimes being able to
> determine if the customer database got out because of a SQL
> injection attack through log examination can be quite
> useful if management is otherwise convinced the problem is
> an insider.. I once spent a few happy weeks poring through
> 40 gigs of transaction log data (yeah, 3 days' worth...) trying
> to identify traces of a hithertofore unknown DOS attack. At
> stake were a bunch of sysadmins' jobs. It was a very
> intellectually stimulating mission.

What did you use to pour through it? You have to be able to load that
40 gigs of data, or break it up into something semi-coherent, and then
you have to be able to scan it quickly enough to get it done within
the year but not so quick you miss something...

> Well, see, what you'd normally do is actually _think_ about
> the problem a little bit - not just jump into it half-assed.
> Most of the commercial logging tools are aimed at attempting
> to "do everything" but you pay a lot for that - if you actually
> know what you want to do, you can do it for not a whole lot.

Tell me d(&#$#!!! The how is what I'm obviously missing...


>
> > - we ever passed and we still got
> >hacked because we didn't hire a new engineer to review the data
> >streaming out of the system and therefore see the new exploit in time
> >to shut it down.
>
> If you are stupid about how you deploy technology, you
> will usually get stupid results. Try explaining that to your
> boss. No - wait - don't.

Yeah, that would be a mistake.

I don't want to be stupid about it, but outside of this list, you
don't hear anything but the marketing buzz on the latest "device" to
make the world a safer, happier place (and NSA compliant).
_______________________________________________
firewall-wizards mailing list
firewall-wizards@xxxxxxxxxxxxxxxxxx
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards

Relevant Pages

  • Re: Thoughts on Logical Log use requested
    ... The onstat -l was taken today, the list of log times for Sunday follow that. ... Physical Logging ... So, you HAVE to have 51,250 minute transactions in an unbuffered logging ... Each transaction is using roughly 1k of log space ...
    (comp.databases.informix)
  • Re: The Poor 5 Cent Piece (Australia)
    ... Unquestionably rendered almost useless by inflation, their only value appears to be when tendering correct change in transactions like $5.05. ...
    (rec.collecting.coins)
  • Re: Replication in databases
    ... It's physical logging, writing modified ... hamsterdb is good for embedded devices, ... concurrency and really bad support for transactions. ... months ago i started writing hamsterdb2, which has a high level of ...
    (comp.databases.theory)
  • Re: No Logging for DTS Batch
    ... Allan Mitchell MCSE,MCDBA, (Microsoft SQL Server MVP) www.SQLDTS.com - The site for all your DTS needs. ... I have the import> working in DTS, but I DO NOT WANT IT TO LOG THE> TRANSACTIONS. ... > Alter table to add keys and clean data. ... > Where and how do I put in a commant to not log anything> in this transaction, yet maintain logging on any other> table in the database. ...
    (microsoft.public.sqlserver.dts)
  • Re: [fw-wiz] RE: IDS (was: FW appliance comparison)
    ... >> That's the dumbest argument against logging I've ever heard. ... We're a transactions ... But, on the bright side, our 2k IDS system did ...
    (Firewall-Wizards)