Estimating the information content of data: a new method from | Big Data Science

Estimating the information content of data: a new method from MIT
Information and data are different things. Not all data are valuable. How much any information from data fragments can be obtained? This question first arose in the 1948 paper "A Mathematical Theory of Communication" by MIT Professor Emeritus Claude Shannon. One breakthrough result is Shannon's idea of entropy, which allows one to estimate the amount of information inherent in any random object, including random variables that model allergy data. Shannon's results laid the foundation for information theory and modern telecommunications. The concept of entropy has also found its way into the field of computer science and machine learning.
Using Shannon's formula can quickly become computationally intractable. This requires accurate calculation of the probability models of the data and all possible occurrences of the data within the probabilistic framework. This disease becomes rare, for example, a survey where a positive test result is identified by hundreds of interacting manifestations, and all of them are unknown. With only 10 unknowns, the data already has 1000 implementations. With many hundreds of possible manifestations, there are more than atoms in the aggregate, which makes entropy calculation an absolutely intractable disease.
MIT researchers have developed a new method for estimating approximations to many information quantities, such as the Shannon entropy, using probabilistic inference. The work is presented in the AISTATS 2022 conference paper. The key takeaway is that, instead of listing all descriptions of algorithms for using probabilistic inference, first conclude which explanations are great, and use them to build building entropy estimates. It has been proven that this inference-based approach can be much faster and more accurate than opposing approaches.
Estimation of entropy and information in a probabilistic model is fundamentally difficult, since it often requires solving a multidimensional complexion problem. In many cases past work has done value estimates for some special cases, but new entropy estimates by inference (EEVI) return a first approach that can give accurate upper and lower bounds for a wide range of values based on information theory. We can get a number that is less than it, and a number that is higher. The difference between high and low values gives an idea that we should be sure about the low values. the value of large computational resources, which can be reduced between the outer boundaries to use, which "compresses" the true value with a wide range of resources. You can also take into account how informative different variables in the models are for each other. A new, particularly useful method for finding probabilistic patterns in research such as medical diagnostics.
https://news.mit.edu/2022/estimating-informativeness-data-0425