RE: Value of "richer" signatures?

From: Ken Arora (kenarora@olympus-mons.net)
Date: 06/24/02 

From: "Ken Arora" <kenarora@olympus-mons.net>
To: "D G" <d_g3377@yahoo.com>, "Robert Graham" <robert_david_graham@yahoo.com>, "Ken Arora" <kenarora@olympus-mons.net>, <focus-ids@securityfocus.com>
Date: Sun, 23 Jun 2002 17:36:41 -0700

DG,

  Thanks for the input. In general, this is what my suspicion
was, but I hadn't seen it spelled out in the literature I
read. The academic reports may be a bit behind the time,
and it was hard to get past the techno-marketing spiel on
most vendor's websites (ie. "stateful packet analysis",
without spelling out how much protocol decode, what protocols
supported, etc..)

  BTW, my intuition was based on reducing the number of
false positives (like the e-mail example you give). Robert
had pointed out that it will be faster; do you agree with
that? Is it that much faster to do "protocol parsing" than
look for strings in the packet/stream?

  Also, what about fancier signature specification in
general (ie. reg-exp) as opposed to just for hard-coded
protocol decode?

Thanks,
--ken

> -----Original Message-----
> From: D G [mailto:d_g3377@yahoo.com]
> Sent: Friday, June 21, 2002 7:11 PM
> To: Robert Graham; Ken Arora; focus-ids@securityfocus.com
> Subject: Re: Value of "richer" signatures?
>
>
> I have to agree with Robert here. I've used ISS,
> Snort, Dragon, and NFR, and I can tell you that they
> all do things a little different (some of them more
> different than others). Many of these IDS's are doing
> very advanced protocol analysis, and state analysis in
> their signatures.
>
> Here's an example of how the newer IDS signatures help
> eliminate false positives through protocol analysis.
> This particular one came from NFR's recent SMTP
> package:
>
> Let's say you are using a simple packet grepping IDS
> with a string match signature to look for people
> probing your smtp servers by sending VRFY: commands to
> see which users exist. Maybe something like
> VRFY:ROOT. Both the packet grepper, and the stateful
> IDS, doing protocol analysis, would probably trigger
> on this. But, let's I send you this email, and you
> have a simple packet grepping engine. Your packet
> grepping IDS may have just false positived, while the
> stateful IDS systems would recognize that VRFY:ROOT
> was really just part of the body, and not an actual
> command. I have not seen RS7.0 yet, but I'm assuming
> this is the type of stuff Robert is talking about.
>
> The other advantage of a stateful IDS is that they can
> track the server response, and tell you how the server
> reacted to the attack: i.e. was it successful or not.
> So, if somebody tries to GET /cgi-bin/test-cgi and
> the server responds with a 404 error, the stateful IDS
> can show you that response.
>
> There are hundreds of examples of why protocol
> analysis is so important in an IDS engine. So, to
> make a statement that all IDS's use simple packet
> grepping techniques makes me think that the articles
> you're reading are about 2 years old. Go talk to
> companies like NFR, ISS, Stealthwatch, Intrusion.com,
> etc. Even within each of those companies, they all do
> protocol analysis and stateful inspection differently.
>
>
> -dg
>
>
>
>
> --- Robert Graham <robert_david_graham@yahoo.com>
> wrote:
> > --- Ken Arora <kenarora@olympus-mons.net> wrote:
> > > I'm just getting to learn about the IDS space, and
> > have looked at some
> > > the exploits and Snort/Arachnid signatures to
> > identify them. While I was
> > > researching, I was slightly surprised to see that
> > most of the IDS's out
> > > there used simple string matches for signatures
> > (at least in the payload)
> > > There were a few efforts to try and handle more
> > complex signatures (some
> > > companies talk of "regular expression" level of
> > signatures).
> > >
> > > I wanted to find out if you (especially those of
> > you who come up with such
> > > signatures) thought that such sophisticated
> > signature vocabularies were
> > > useful/worth pursuing? Looking at some of the
> > exploits, I can see where
> > > finding a string "in context" (context might be
> > "these two string on the
> > > same line" or "this string in the context of a GIF
> > comment field" or ...)
> > > might be useful, at least to eliminate false
> > positives, maybe as the only
> > > way of dealing with more subtle attacks?
> > >
> > > If more of the IDS software out there could do
> > this, would we see more
> > > advanced signatures? (is this just a
> > chicken-and-egg problem?) How
> > > useful would they be, in the real world?
> >
> > The way you phrase this makes me think that this is
> > a troll of some sort,
> > because this is exactly the argument we use at ISS.
> > "Context" is what
> > differentiates ISS RealSecure 7 from the rest of the
> > IDSs on the market.
> >
> > Context is important. I would encourage you to look
> > at the difference between
> > Snort/1.8.6 and Snort/1.9 (you can get the source to
> > Snort whereas you can't
> > get our source to RealSecure). Snort/1.9 adds "TCP
> > connection state" context.
> > For example, consider two TCP connections where the
> > "client" side picks a
> > special port number.
> > (A) from client port 21 to server port 80 (this is
> > an HTTP connection)
> > (B) from client port 80 to server port 21 (this is
> > an FTP connection).
> > Now consider when the IDS sees a packet from port 21
> > to port 80. If this packet
> > is in the context of connection A, the this is an
> > "HTTP request" packet. If
> > this packet is seen in the context of connection B,
> > then this is an "FTP
> > response" packet. The only way to know
> > which-is-which is to look at the state
> > of the connection.
> >
> > Snort versions prior to 1.9 get confused by this.
> > Snort v1.9 does not.
> >
> > I am the chief architect for ISS RealSecure where we
> > have taken this "context"
> > to a whole new level. We do full real-time
> > protocol-decoding of more than 60
> > application-layer protocols.
> >
> > For example, you might have an FTP server that is
> > attempting to hack FTP
> > clients. When user does an "mget *" command, the
> > client first gets a directory
> > listing, then gets each file in that listing. The
> > listing itself as well as the
> > files are sent across different TCP connections from
> > the original FTP
> > command-channel on port 21. A hostile server could
> > return a listing including a
> > filename with "../..": a classic directory-climbing
> > attack.
> >
> > The way that RealSecure works is that it does a
> > protocol-decode of the FTP
> > command channel and figures out which TCP
> > connections are FTP data connections.
> > It then figures out which ones are "file transfers",
> > and which ones are
> > transfers of other information, such as "directory
> > listings". When it sees a
> > directory listing, it decodes that information to
> > see if there is a "../.."
> > pattern.
> >
> > Application-layer protocol decodes are expensive in
> > terms of CPU, but the
> > benefit is that you don't have to look for
> > signatures outside of appropriate
> > context, which speeds things up overall. In the
> > above FTP example, think about
> > the cost of searching for "../.." in every packet
> > vs. the cost of searching for
> > it just in FTP directory listings. In a fully loaded
> > 100-mbps serving FTP
> > traffic, you might get 20,000 packets/second of FTP
> > file data, but a directory
> > listing happens once every few seconds.
> >
> > The downside of this is that it isn't easily
> > extensible. We have to write raw C
> > code for this. The Snort ruleset is far more
> > flexible and easier to use by the
> > end-user and "collaborating" on signatures. Snort
> > has a "plugin" interface and
> > preprocessors where you can do some of this (e.g.
> > uricontent), but the
> > technique is generally not used. Conversely, while
> > RealSecure has user-defined
> > signatures where users can look for patterns within
> > protocol-context, customers
> > really prefer to use the more flexible "Trons"
> > (Snort ruleset clone) module in
> > RealSecure in order to write their own signatures.
> >
> > BTW, you can't look at a single IDS (e.g. Snort) and
> > make assumptions about the
> > rest of the field of IDS products: they are all very
> > different from each other.
> > I read a lot of papers on IDS that make sweeping
> > generalizations about the IDS
> > market that really only refer to Snort (because
> > Snort is the only one they can
> > get source for). Snort's and RealSecure's detection
> > methods are utterly
> > different (curiously, though, we both have TCP
> > connection state whereas most
> > other ones don't; it's interesting how Snort/1.9 and
> > RealSecure/7 have both
> > moved closer to each other).
> >
> > Robert Graham
> > Chief Architect, ISS
> >
> >
> >
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Yahoo! - Official partner of 2002 FIFA World Cup
> > http://fifaworldcup.yahoo.com
>
>
> __________________________________________________
> Do You Yahoo!?
> Yahoo! - Official partner of 2002 FIFA World Cup
> http://fifaworldcup.yahoo.com
>

Quantcast