Mutual Information as probability

Could the mutual information over the joint entropy:
$$
0 \leq \frac{I(X,Y)}{H(X,Y)} \leq 1$$

be defined as:”The probability of conveying a piece of information from X to Y”?

I am sorry for being so naive, but I have never studied information theory, and I am trying just to understand some concepts of that.

Answer

The measure you are describing is called Information Quality Ratio [IQR] (Wijaya, Sarno and Zulaika, 2017). IQR is mutual information $I(X,Y)$ divided by “total uncertainty” (joint entropy) $H(X,Y)$ (image source: Wijaya, Sarno and Zulaika, 2017).

enter image description here

As described by Wijaya, Sarno and Zulaika (2017),

the range of IQR is $[0,1]$. The biggest value (IQR=1) can
be reached if DWT can perfectly reconstruct a signal without losing of
information. Otherwise, the lowest value (IQR=0) means MWT is not
compatible with an original signal. In the other words, a
reconstructed signal with particular MWT cannot keep essential
information and totally different with original signal
characteristics.

You can interpret it as probability that signal will be perfectly reconstructed without losing of information. Notice that such interpretation is closer to subjectivist interpretation of probability, then to traditional, frequentist interpretation.

It is a probability for a binary event (reconstructing information vs not), where IQR=1 means that we believe the reconstructed information to be trustworthy, and IQR=0 means that opposite. It shares all the properties for probabilities of binary events. Moreover, entropies share a number of other properties with probabilities (e.g. definition of conditional entropies, independence etc). So it looks like a probability and quacks like it.


Wijaya, D.R., Sarno, R., & Zulaika, E. (2017). Information Quality Ratio as a novel metric for mother wavelet selection. Chemometrics and Intelligent Laboratory Systems, 160, 59-71.

Attribution
Source : Link , Question Author : luca maggi , Answer Author : Tim

Leave a Comment