Re: Administrivia #35466 - NetCraft IIS insecurity statistics

From: Eduardo Subelman (subelman@ITGSSI.COM)
Date: 11/08/01 

Message-ID:  <3BEA4F17.28503.56B6DD@localhost>
Date:         Thu, 8 Nov 2001 09:23:35 -0800
From: Eduardo Subelman <subelman@ITGSSI.COM>
Subject:      Re: Administrivia #35466 - NetCraft IIS insecurity statistics
To: NTBUGTRAQ@LISTSERV.NTBUGTRAQ.COM

Although some of your points are well taken, your discussion of
Netcrafts statistics leaves much to be desired.

On 8 Nov 2001, at 9:40, Russ wrote:

> 2. NetCraft performs "several hundred" Web Server Security evaluations
> per month, and this, and only this, makes up the dataset for their Web
> Server Security statistics. "Several hundred" out of 9.6+ million is
> hardly statistically significant, especially since there's no attempt
> to ensure that the subset is representative of the main dataset.

Several hundred (say two to three hundred) is indeed statistically
significant. Statistical significance (or lack thereof) is
independent of the size of the target population. You seem to be
under the mistaken impression that for a sample to be significant it
has to contain a certain proportion of the whole population. This is
false, as any textbook in Statisics will tell you.

> 3. Of the "several hundred" Web Server Security evaluations that
> NetCraft does monthly, and uses for their Web Server Security
> statistics, "In general, around 60% of all the sites we test are
> running IIS". So, of the ~200-300 sites tested per month, only
> ~120-180 of them are running IIS. Ergo, only ~120-180 tests are used
> to extrapolate statistics over a group of 9.6+ million sites.

Even taking your low estimate for what "several hundred" means, you
conclude that about 120 sites using IIS were tested, yielding 11%
infected. With this information, I can still say (using basic
Statistical Theory) that "I am 90% confident that the true percentage
of infected sites is between 7% and 15% (11 +/- 4). This means that
between 672,000 and 1,440,000 IIS sites are wide open. And I'm 90%
sure of that.

> NetCraft is normally pointed to frequently as a reliable source of
> statistical data. Unfortunately, now that they are in the Security
> Testing business, they've seen fit to use a ridiculously small test
> set to hype the insecurity of IIS.

I don't think it is "ridiculously" small. Even if they went to a
sample of 1000, their margin of error would still be 11 +/- 2%, and
their costs would go up ten-fold for that small gain in accuracy.

 +-------------------------+-----------------------------------+
 | Eduardo Subelman, Ph.D.| Investment Technology Group |
 | subelman@itgssi.com | 400 Corporate Pointe, Suite 600 |
 | 213.270.7920 (voice) | Culver City, CA 90230 |
 | 310.216.0933 (fax) | http://www.itginc.com |
 +-------------------------+-----------------------------------+

============================================================================
Delivery co-sponsored by Trend Micro, Inc.
============================================================================
BEST-OF-BREED ANTIVIRUS SOLUTION FOR MICROSOFT EXCHANGE 2000
Earn 5% rebate on licenses purchased for Trend Micro ScanMail for
Microsoft Exchange 2000 between October 1 and November 16. ScanMail
ensures 100% scanning of inbound and outbound traffic and provides
remote software management. For program details or to download your
30-day FREE evaluation copy:
http://www.antivirus.com/banners/tracking.asp?si=53&bi=245&ul=http://www.a
ntivirus.com/smex2000_rebate

Relevant Pages

  • Administrivia #35466 - NetCraft IIS insecurity statistics
    ... Numerous recent articles have used information provided by NetCraft, ... Microsoft IIS site security. ... Statistics, and stories, like these are what drive people away from IIS. ... NetCraft's Web Server Survey statistics, 33+ million web servers on the ...
    (NT-Bugtraq)
  • RE: IIS
    ... If you don't know any other way to harden IIS then patching it you ... > I recently read a statistic that said apache is hacked more than IIS ... and I have also seen statistics go the other way. ... secure as the administrator of that system is knowledgeable. ...
    (Security-Basics)
  • Monitoring IIS subwebs
    ... I'm interested in monitoring a little basic information on subwebs under IIS. ... Things like number of hits per day, and a handful of other statistics. ...
    (microsoft.public.dotnet.languages.vb)
  • Re: IIS logging analyse per page
    ... > I am looking for a tool to analyse the logging created by IIS. ... > The statistics should preferably be shown on a webpage. ...
    (microsoft.public.inetserver.iis)
  • Re: Medical Research
    ... expected the probability that this type of treatment would ... BTW, the 7 out of 8 is statistically significant, but I ... Herman Rubin, Department of Statistics, Purdue University ...
    (talk.politics.medicine)