Re: issues with statistical test suite from http://csrc.nist.gov/rng/

From: Cristiano (cristiano.pi_at_NSquipo.it)
Date: 01/16/04 

Date: Fri, 16 Jan 2004 20:03:56 GMT

Luke Kenneth Casson Leighton wrote:
> the alternative explanation is of course that the diehard cdrom data,
> http://stat.fsu.edu/pub/diehard/cdrom/ is not "true" random data.

Or that your implementation is bad.

> - FFT test in anything less than 1e6 sample sizes
>
> a test on the minimum recommended length of 1000 or a
> anything up to around 1,000,000 bits, shows non-uniform
> distribution of p-values on what is purported to be
> "true" random data.
>
> on a test of length 1,000, the C8 histogram bucket,
> representing p-values of between 0.7 and 0.8, was empty!

Really!? I suggest to double check your implementation!
Running that test 100 times I get a very good distribution; all the bins
have about 10 p-values.

> - Rank test on anything less than 1e6 sample sizes
>
> a test of 100,000 sample sizes on 8,000 runs gives a
> skewed histogram of p-values - sufficient for the data
> to be determinined as non-random.

Also in this case I get the p-values: perfectly skewed (-0,031), correctly
binned and evenly distributed.

> - Lempel-Ziv test on anything.
>
> this one is particularly weird: on sample sizes of 1e6,
> you should not be looking for a uniform distribution of
> p-values but instead should be looking for a small percentage
> of extra p-values in the C4 (0.3 - 0.4) histogram bucket!

I don't know what kind of code you are using. You should get p-values from 0
to 0.5 because they said "The test is preferably one-sided" so they use 1/2
erfc(...). But we need a p-values from 0 to 1, so you must throw away 1/2.
This way I get a good distribution (with good generators).

> in short, stay well clear of using this test.

In short: you have not read the NIST paper, you have not checked your
implementation against the test vectors and you don't know what you're
saying, so you should avoid this kind of claims.

Cristiano

Relevant Pages