Log probability vs product of probabilities

According to this wikipedia article, one can represent the product of probabilities x⋅y as -log(x) - log(y) making the computation more computationally optimal. But if I try an example say:

p1 = 0.5
p2 = 0.5
p1 * p2 = 0.25
-log(p1) - log(p2) = 2

p3 = 0.1
p4 = 0.1
p3 * p4 = 0.01
-log(p3) - log(p4) = 6.64

The product of probabilities p1 and p2 is higher then the one of p3 and p4, but the log probability is lower.

How come?


I fear you have misunderstood what the article intends. This is no great surprise, since it’s somewhat unclearly written. There are two different things going on.

The first is simply to work on the log scale.

That is, instead of “$p_{AB} = p_A\cdot p_B$” (when you have independence), one can instead write “$\log(p_{AB}) = \log(p_A)+ \log(p_B)$”. If you need the actual probability, you can exponentiate at the end to get back $p_{AB}$: $\qquad p_{AB}=e^{\log(p_A)+ \log(p_B)}\,,$ but if needed at all, the exponentiation would normally be left to the last possible step. So far so good.

The second part is replacing $\log p$ with $-\log p$. This is so that we work with positive values.

Personally, I don’t really see much value in this, especially since it reverses the direction of any ordering ($\log$ is monotonic increasing, so if $p_1<p_2$, then $\log(p_A)< \log(p_2)$; this order is reversed with $-\log p$).

This reversal seems to concern you, but it’s a direct consequence of the negation – it should happen with negative log probabilities. Think of negative log probability as a scale of “rarity” – the larger the number, the rarer the event is (the article refers to it as ‘surprise value’, or surprisal, which is another way to think about it). If you don’t like that reversal, work with $\log p$ instead.

To convert negative-log-probabilities back to probabilities, you must negate before exponentiating. If we say $s_i = -\log(p_i)$ ($s$ for ‘surprise value’), then $p_{AB}=e^{-[s_A+ s_B]}\,.$ As you see, that reverses direction a second time, giving us back what we need.

Source : Link , Question Author : spacemonkey , Answer Author : gung – Reinstate Monica

Leave a Comment