For an arbitrary continuous random variable, say X, is its differential entropy always less than ∞? (It’s ok if it’s −∞.) If not, what’s the necessary and sufficient condition for it to be less than ∞?
I thought about this question some more and managed to find a counter-example, thanks also to the Piotr’s comments above. The answer to the first question is no – the differential entropy of a continuous random variable (RV) is not always less than ∞. For example, consider a continuous RV X whose pdf is
It’s not hard to verify that its differential entropy is infinite. It grows quite slowly though (approx. logarithmically).
For the 2nd question, I am not aware of a simple necessary and sufficient condition. However, one partial answer is as follows. Categorize a continuous RV into one of the following 3 Types based on its support, i.e.
Type 1: a continuous RV whose support is bounded, i.e. contained in [a,b].
Type 2: a continuous RV whose support is half-bounded, i.e. contained in [a,∞) or (−∞,a]
Type 3: a continuous RV whose support is unbounded.
Then we have the following –
For a Type 1 RV, its entropy is always less than ∞, unconditionally.
For a Type 2 RV, its entropy is less than ∞, if its mean (μ) is finite.
For a Type 3 RV, its entropy is less than ∞, if its variance (σ2) is finite.
The differential entropy of a Type 1 RV is less than that of the corresponding uniform distribution, i.e. log(b−a), a Type 2 RV, that of the exponential distribution, i.e. 1+log(|μ−a|), and a Type 3 RV, that of the Gaussian distribution, i.e. 12log(2πeσ2).
Note that for a Type 2 or 3 RV, the above condition is only a sufficient condition. For example, consider a Type 2 RV with f(x)=3x2
for x>3. Clearly, its mean is infinite, but its entropy is 3.1 nats. Or consider a Type 3 RV with f(x)=9|x|3
for |x|>3. Its variance is infinite, but its entropy is 2.6 nats. So it would be great, if someone can provide a complete or more elegant answer for this part.