Is differential entropy always less than infinity?

For an arbitrary continuous random variable, say X, is its differential entropy always less than ? (It’s ok if it’s .) If not, what’s the necessary and sufficient condition for it to be less than ?

Answer

I thought about this question some more and managed to find a counter-example, thanks also to the Piotr’s comments above. The answer to the first question is no – the differential entropy of a continuous random variable (RV) is not always less than . For example, consider a continuous RV X whose pdf is
f(x)=log(2)xlog(x)2
for x>2.

It’s not hard to verify that its differential entropy is infinite. It grows quite slowly though (approx. logarithmically).

For the 2nd question, I am not aware of a simple necessary and sufficient condition. However, one partial answer is as follows. Categorize a continuous RV into one of the following 3 Types based on its support, i.e.

Type 1: a continuous RV whose support is bounded, i.e. contained in [a,b].
Type 2: a continuous RV whose support is half-bounded, i.e. contained in [a,) or (,a]
Type 3: a continuous RV whose support is unbounded.

Then we have the following –

For a Type 1 RV, its entropy is always less than , unconditionally.
For a Type 2 RV, its entropy is less than , if its mean (μ) is finite.
For a Type 3 RV, its entropy is less than , if its variance (σ2) is finite.

The differential entropy of a Type 1 RV is less than that of the corresponding uniform distribution, i.e. log(ba), a Type 2 RV, that of the exponential distribution, i.e. 1+log(|μa|), and a Type 3 RV, that of the Gaussian distribution, i.e. 12log(2πeσ2).

Note that for a Type 2 or 3 RV, the above condition is only a sufficient condition. For example, consider a Type 2 RV with f(x)=3x2
for x>3. Clearly, its mean is infinite, but its entropy is 3.1 nats. Or consider a Type 3 RV with f(x)=9|x|3
for |x|>3. Its variance is infinite, but its entropy is 2.6 nats. So it would be great, if someone can provide a complete or more elegant answer for this part.

Attribution
Source : Link , Question Author : syeh_106 , Answer Author : syeh_106

Leave a Comment