Today I ran across the book “Information theory: A tutorial introduction” by James Stone and thought for a moment or two about the

extent of useof information theory inapplieddata science(if you’re not comfortable with this still somewhat fuzzy term, thinkdata analysis, which IMHO data science is a glorified version of). I’m well aware of the significant use ofinformation theory-basedapproaches,methodsandmeasures, especiallyentropy,under the hoodof various statistical techniques and data analysis methods.However, I’m curious about the

extent/levelof knowledge that is needed for anapplied social scientistto successfullyselectandapplythose concepts, measures and tools without diving too deep into mathematical origins of the theory. I look forward to your answers, which might address my concern within the context of the above-mentioned book (or other similar books – feel free to recommend) or in general.I would also appreciate some recommendations for print or online sources that discuss

information theoryand its concepts, approaches, methods and measures in thecontextof (incomparisonwith) other (more)traditional statistical approaches(frequentistandBayesian).

**Answer**

So the first part of question: *Do data scientists need to know information theory*? I thought the answer is no until very recently. The reason I changed my mind is one crucial component: noise.

Many machine learning models (both stochastic or not) use noise as part of their encoding and transformation process and in many of these models, you need to infer the probability which the noise affected after decoding the transformed output of the model. I think that this is a core part of information theory. Not only that, in deep learning, KL divergence is a very important measure used that also comes from Information Theory.

Second part of the question: I think the best source is David MacKay’s Information Theory, Inference and Learning Algorithms. He starts with Information Theory and takes those ideas into both inference and even neural networks. The Pdf is free on Dave’s website and the lectures are online which are great

**Attribution***Source : Link , Question Author : Aleksandr Blekh , Answer Author : Nick Cox*