Russ Cooper's AusCERT Presentation on MS Security Bulletins
From: Russ (Russ.Cooper_at_RC.ON.CA)
Date: Wed, 2 Jun 2004 13:43:22 -0400 To: NTBUGTRAQ@LISTSERV.NTBUGTRAQ.COM
As you may have heard, I did a presentation last week at the 2004
AusCERT Conference in Gold Coast, Australia.
My presentation was the culmination of analysis I performed on all
Microsoft Security Bulletins published by Microsoft from January 1, 2000
to date. I analyzed the *vulnerabilities*, dissecting each bulletin into
their respective vulnerabilities. As we all know, each bulletin MS
produces may involve numerous vulnerabilities. In addition,
vulnerabilities addressed by a bulletin may affect some versions but not
others. I tabulated all of this information based on the facts in the
The purpose of this was to address a common problem I see in the media,
namely, attempting to use the count of bulletins in comparisons with
other OS'. Patch count comparisons serve no purpose, but counting
patched vulnerabilities, IMO, is a more accurate comparison. In
addition, I grouped vulnerability counts according to whether they'd
likely affect desktop, server, and IIS servers. Again, comparing raw
numbers for "Windows" against other OS' isn't a correct comparison
either, so using numbers based on a role made more sense to me.
My presentation made no attempt to compare Windows to any other OS. I
compared MS to MS, NT 4.0 Workstation versus W2K Professional versus
Windows XP, NT 4.0 Server versus W2K Server versus W2K3. I compared IE
5.0x versus IE 5.5 versus IE 6.0. For all, I made no distinction wrt
Service Packs (and therefore if security vulnerabilities were addressed
by Service Packs alone, they were not considered.)
Again, I'd like to stress, my analysis did not consider many things
which are important in determining the security of a system. For
example, I did not address severity, either as designated by MS or by me
nor did I address exploitability. Although my numbers did not include
vulnerabilities which were not installed by default, they did include
vulnerabilities which could have been avoided by configuration or by
removing components. I did exclude all vulnerabilities for things like
SQL (including MSDE), Exchange, IIS add-ons (like Media Server) and
other server products.
I analyzed 452 vulnerabilities in 298 Microsoft Security Bulletins. I
found very little difference in the number of vulnerabilities affecting
different versions of a given product. The vast majority of
vulnerabilities affected all of the versions of a product I looked at,
meaning to me that the vulnerabilities were in legacy code carried
forward into new versions. That's the sort of stuff the Security Push
was supposed to find, IMO, yet there was nothing in my analysis to show
the Push had an impact on this problem.
In the period I covered, here are some of the stats I discussed;
1. Comparing Desktop OS' without browsers
NT 4.0 Workstation = 68 vulnerabilities
Windows 2000 Professional = 86 vulnerabilities
Windows XP = 55 vulnerabilities
NT 4.0 Server = 76 vulnerabilities
Windows 2000 Server = 98 vulnerabilities
Windows Server 2003 = 24 vulnerabilities
2. Comparing the first 300 days of Windows 2000 Server to Windows Server
Windows 2000 Server = 27 vulnerabilities
Windows Server 2003 = 24 vulnerabilities
3. Comparing Windows NT 4.0 Server to Windows Server 2003 during the
first 300 days of Windows Server 2003
Windows NT 4.0 Server = 22 vulnerabilities
Windows Server 2003 = 24 vulnerabilities
This is where I made some comments about NT and Microsoft shareholders.
What I said was this. Some people think they should wait for an SP1
before they deploy a product. Considering that NT 4.0 Server was at SP6
prior to the beginning of my 52 month analysis, if you waited for SP6 of
W2K3 before deploying you may only experience an 8.333% reduction in the
number of vulnerabilities which affect it. IOWs, if you think that
waiting for a product to mature is going to make it less vulnerable,
you're probably waiting forever.
I also made the point that since newer versions aren't significantly
less vulnerable, people aren't going to upgrade to them to be less
vulnerable. That may be something MS shareholders might be concerned
about, if they believe the desire to be less vulnerable is a purchase
motivator. The actual bulletin point in the presentation was;
"If Microsoft made money from upgrades, you'd think these numbers might
bother some people"
4. I then did a comparison of desktop OS' where the browsers were kept
up-to-date, revised to new versions as they were released. It showed, to
me, that this tactic didn't significantly reduce vulnerabilities either.
Again, largely because vulnerabilities affected all versions.
5. I then did a comparison of NT 4.0 desktops, one which stuck with IE
5.01 throughout the time, another which upgraded and stuck with IE 5.5,
and another which upgraded to each new IE version. Overall, the one
which stuck with IE 5.01 had less vulnerabilities, but even so, it was
only 7% less vulnerable than the one which upgraded to each new version.
6. I made the same comparison using a Windows 2000 Professional and
found almost identical differences (albeit there were more
vulnerabilities applicable to W2K than NT 4.0).
I then had a slide that simply said;
"Older is clearly better!"
Throughout my presentation the lowest number of vulnerabilities always
applied to NT 4.0. My point here was that new features meant new
vulnerabilities, either because new features were introduced which had
new vulnerabilities, or, new versions meant a new examination by
vulnerability discoverers. In retrospect, chances are the reason new
versions means new vulnerabilities is because of new examination, since
by and large vulnerabilities apply to all versions.
In the context of Microsoft's security efforts, I made comments about
the fact that the newer versions had more vulnerabilities than the older
versions. Despite the fact the difference is extremely small, fact is
that its not getting better, something which cannot be ignored by
analysis of the facts.
7. I then compared IIS versions. Given the timeframe of the products,
the numbers are very different;
IIS 4.0 = 231 vulnerabilities
IIS 5.0 = 282 vulnerabilities
IIS 6.0 = 60 vulnerabilities
I went on to say that in the period since W2K3's release, IIS 6.0 boxes
were 11% less vulnerable than W2K IIS 5.0 servers. This, however, IMO
was largely due to configuration and not a lack of vulnerable code. I
said that if you configured any IIS box the way W2K3 IIS 6.0 was
configured and you'd get roughly the same security. IOWs, where were the
results of the Security Push? Surely the results weren't only a new
8. My last graph depicted the number of vulnerabilities versus the
number of patches. IOWs, does the fact that there have been fewer
patches of late mean there have been fewer vulnerabilities. The graph
indicated that the answer is no, not really. At various points in time
over the past 52 months there have been ups and downs, and the last 6
months is no different.
9. My concluding slides were all titled "Patch-O-Mania". I started by
stating that I believed patching was not a reason to upgrade to a new
product. IOWs, if you thought you'd upgrade to a new product because it
would mean you wouldn't have to patch so often, you're wrong. I
questioned whether or not an 11% reduction in the vulnerability of W2K3
represented a start to Microsoft's Security Push. Personally, I don't
believe that's a significant enough figure to say it really is.
I then said the Security Push is about consumers, not corporate users. I
believe the MS Security Bulletins are designed for home users, not for
corporate users. The whole idea of patching is primarily intended, again
IMO, for home users. I mentioned how I believed that XP SP2 was
excellent, for home users, and that it would enable consumers to be less
aware of "Patch-O-Mania" because automatic updates would be enabled.
Since I was in Australia, and since there people pay for bandwidth above
an ~50MB cap, I did point out however that XPSP2 was 276MB and its
adoption is required for us to reap its benefits.
I concluded with Corporations. TruSecure's Sasser survey showed that
unless you achieved 100.0% patching you were in worse shape than if you
didn't try patching at all. I pointed out that none of Microsoft's patch
deployment products could guarantee 100.0% effective patch deployment,
nor could any 3rd party product. Ergo, Corporations have no way of
knowing whether they've achieved the 100.0% patch deployment. I
questioned how the Security Push could think that patching was an
effective part of its efforts given these facts.
So, in the end, I wasn't recommending that people stick with NT 4.0, nor
was I saying that newer products are more exploitable or less secure
than older ones. In the numerous media interviews I gave throughout the
region over the days following the event I repeatedly told reporters
this. I do believe that we have received insufficient benefits from the
Microsoft Security Initiative, but stressed that XP SP2 represented very
significant changes in the way MS approaches security. I stressed that
it remained to be seen whether or not MS will be able to apply the new
principles contained within XP SP2 to Corporate users, but if they can
it would be great.
I called for a new SP for Windows NT 4.0, considering the fact its still
widely in use and has been more than 4 years without one yet has had at
least 68 vulnerabilities patched.
I stressed that, IMO, far too much effort is being placed on patching IE
vulnerabilities. To the best of my knowledge, only 2 wide-spread attacks
have occurred involving IE vulnerabilities, yet there have been at least
83 vulnerabilities patched for IE. Clearly a lot of effort is being
spent patching vulnerabilities which have not resulted in exploits, IMO,
a large waste of Corporate resources.
Finally, without making a sales pitch, I tried to stress that there are
many ways to mitigate against vulnerabilities. TruSecure Corporation, my
employer, and I, firmly believe that patching is amongst the least
effective methods. For example, through our knowledge transfer annual
subscription Risk Management Programs, we helped our customers stay
secure despite only recommending 3 Microsoft patches be applied urgently
last year. For more information about those services, see;
The bottom line is that we, TruSecure Corporation, and I, are firm
believers in Microsoft Products. Anyone who's been a subscriber of
NTBugtraq for any length of time should realize this. We and Microsoft
may differ on the best approaches to stay secure, or on where Microsoft
should put its efforts or priorities, we remain a committed Microsoft
Partner. Neither TruSecure nor Microsoft have asked me to say this, I
just feel that with so many media reports making me out to be a vehement
Microsoft detractor, clarification was needed.
Russ - TruSecure Corporation Senior Scientist/NTBugtraq Editor
(note: I've wanted to drop the "Surgeon General" moniker for 2 years
now. I dislike "Senior Scientist" almost as much considering I never
finished High School. Suggestions for a new corporate title for me are
Patch Automation v6.0 by Mobile Automation, Inc. allows you to quickly
identify and fix all PC's that are exposed to the Sasser worm! Our
solution provides quick and seamless discovery and deployment of all your
PC computer's Microsoft security patching needs. Regardless of where
you're PC's reside (inside the LAN, at home or on the road), Patch
Automation gets the job done. Contact us to learn about our free 30-day
trial version at 800-344-1150 or visit our website at