Re: Entropy of draw from normal or binomial distribution



On 2011-08-30, Jeffrey Goldberg <nobody@xxxxxxxxxxxx> wrote:
Let me give a specific example of my question first.

Suppose Alice flips a coin 100 times but doesn't record the sequence.
All she has is the number of heads and tails. How many bits of entropy
can we assign to the number of heads that show up?

More generally, how do we calculate the bits of entropy that we can get
from a drawing from a binomial distribution (as in the example above) or
a normal distribution?

-sum_i p_i ln(p_i)
is the usual expression for the entropy of a distribution. So in your
case, p_r= (1/2)^100 100!/(r! (100-r)!)
will be the probability of getting r heads (assuming a fair coin) in 100
tosses.



Note that the distributions need to be modified so that there is only a
finite number of possible results from a draw. (Otherwise the entropy
would be infinite, unless I'm mistaken.)

Nope. Because the probablility drops off rapidly for large values.


So suppose draw integers distributed normally with say, mean 0, and
standard deviation 100. How many bits of entropy do we get out of a
single draw? (I assume that only the s.d. matters. I can't see why the
mean would play a role in the answer.)

I'm happy with answers of RTF[MAPB] (Manual, Article, Paper, Book) as
long as you point me toward those resources.

Cheers,

-j


.



Relevant Pages

  • Re: behavior as mapping
    ... estimating a probability distribution, the distribution ... sequence with equal probability - since you have microsecond temporal ... reduction of the entropy Pto the entropy P ... If there were 4 genes we would need 2 bits of binding site info. ...
    (comp.ai.philosophy)
  • Re: Entropy of draw from normal or binomial distribution
    ... Suppose Alice flips a coin 100 times but doesn't record the sequence. ... All she has is the number of heads and tails. ... How many bits of entropy ... sometimes interested in the "minentropy" (which is essentially the entropy ...
    (sci.crypt)
  • Computational secure entropy extraction
    ... distilling entropy from an unknown distribution. ... there existed some universal entropy distiller that could be used on all input ... D is -secure if given that k is drawn from any distribution ... Let's define that a "hit" is the case ...
    (sci.crypt)
  • Re: Entropy of draw from normal or binomial distribution
    ... All she has is the number of heads and tails. ... how do we calculate the bits of entropy that we can get ... a normal distribution? ... finite number of possible results from a draw. ...
    (sci.crypt)
  • Re: behavior as mapping
    ... estimating a probability distribution, the distribution ... sequence with equal probability - since you have microsecond temporal ... reduction of the entropy Pto the entropy P ... If there were 4 genes we would need 2 bits of binding site info. ...
    (comp.ai.philosophy)