RE: Why eEye Retina (was MBSA scanner)

From: Robert E. Lee (robert_at_dyadsecurity.com)
Date: 04/22/04

  • Next message: Riley Hassell: "RE: Why eEye Retina (was MBSA scanner)" 
    Date: Thu, 22 Apr 2004 11:48:22 -0700
To: "Mike Murray" <mmurray@ncircle.com>

    Mike,

    I agree with you. The skills are indeed separate, but it helps to have
    tools that know about the latest public problems.

    That being said I'm finding myself cursing at every VA tool out there.
    There are four things wrong with all tools that I have played with:

    1) False Positives - No <insert silly tool here>, this machine is not
    vulnerable to the "bitch slap" DoS :). This part takes a long time to
    ferret out. You don't want to tell people they're broken when they're
    not.

    2) False Negatives - Even easy things are often missed by all tools
    (which is why most people end up using 2-3 of them for complete
    results). You can only test what you know about... and hope that you
    wrote your test in a sane universal way.

    3) Too much focus on "network" problems. Not enough "outside of the
    box" thinking here. Granted a lot of this work may be through manual
    steps the tester makes, but there should be a good way of tracking every
    significant activity/discovery the team makes.

    4) Broken AI - It's better to have a security analyst make conclusions
    than a piece of software that can't possibly see the big picture.

    In short, using these tools for a test of 1-50 systems is useful. Once
    you cross the 50-100-1000+++ system mark it really becomes information
    deluge.

    These tools are being sold to non-security professionals in the hopes of
    helping them improve InfoSec posture. This is unfortunate because it's
    missing the mark badly.

    My team is writing/using a data correlation engine that allows for a
    team of testers to have the computer do the tedious work that humans
    make get bored with and are error prone at, while allowing for human
    interaction throughout. It is all based around the OSSTMM
    (www.osstmm.org). This will make a good team of testers a lot more
    efficient in their time and more complete in their analysis.

    This represents a goal/design shift for "VA" tools, but I think it is an
    important one that other companies might follow, especially as they
    realize that companies that can afford $17,000 for VA software are
    likely have an internal team of testers anyway :).

    Robert

    -----Original Message-----
    From: Mike Murray [mailto:mmurray@ncircle.com]
    Sent: Wednesday, April 21, 2004 8:33 PM
    To: Shawn Edwards; clarke-cummings@columbus.rr.com
    Cc: pen-test@securityfocus.com
    Subject: RE: Why eEye Retina (was MBSA scanner)

    Let me state up front: I work for a competitor in the VA market, so I'm
    going to stay far away from any discussion on products, and try to stick
    with a bit of philosophy.

    I had one comment on something that Shawn said:

    > I know for a fact that they have some very skilled persons
    > doing dev there. ... Just check some of their development
    > discoveries that's gotta count for something!

    While this is definitely an argument for the fact that a company has
    very smart people working for it (which is definitely not in question in
    eEye's case), I question the validity of the argument as far as the
    evaluation of a network VA tool. If the ability to discover new
    vulnerabilities were the gold standard for a good VA tool, we'd all be
    buying something that Dave Aitel wrote.

    While it is often given as a reason that one tool is better than
    another, it simply doesn't follow that an aptitude for discovering new
    vulnerabilities in code is the same as an aptitude for discovering known
    vulnerabilities in running services in the real world. IMHO, the skills
    are related, but significantly different.

    In my mind the analogy is similar to that of the difference between
    medical research and surgery. People who practice one extremely well
    don't usually practice the other to the same level, even though the
    skills (though not necessarily the mindsets) required to perform both
    are somewhat similar in many cases. One just happens to be focused on
    discovering new techniques out in the world, and the other happens to be
    focused on saving lives.

    My $0.02.

    M

    -------------------------------------------------
    Michael Murray
    Director of Vulnerability and Exposure Research
    nCircle Network Security
    Office: 416-533-5305
    -------------------------------------------------

    ------------------------------------------------------------------------
    ------
    Ethical Hacking at the InfoSec Institute. Mention this ad and get $545
    off
    any course! All of our class sizes are guaranteed to be 10 students or
    less
    to facilitate one-on-one interaction with one of our expert instructors.
    Attend a course taught by an expert instructor with years of
    in-the-field
    pen testing experience in our state of the art hacking lab. Master the
    skills
    of an Ethical Hacker to better assess the security of your organization.
    Visit us at:
    http://www.infosecinstitute.com/courses/ethical_hacking_training.html
    ------------------------------------------------------------------------
    -------

    ------------------------------------------------------------------------------
    Ethical Hacking at the InfoSec Institute. Mention this ad and get $545 off
    any course! All of our class sizes are guaranteed to be 10 students or less
    to facilitate one-on-one interaction with one of our expert instructors.
    Attend a course taught by an expert instructor with years of in-the-field
    pen testing experience in our state of the art hacking lab. Master the skills
    of an Ethical Hacker to better assess the security of your organization.
    Visit us at:
    http://www.infosecinstitute.com/courses/ethical_hacking_training.html
    -------------------------------------------------------------------------------

  • Next message: Riley Hassell: "RE: Why eEye Retina (was MBSA scanner)"