"The Deterministic Dual to Shannon’s Entropy"

The Bayes Forum connects people on the Munich/Garching campus who apply Bayesian methods and information theory in their research. The focus is on astronomy, astrophysics, and cosmology, but contributions from other disciplines are welcome. Typical topics include model fitting and evaluation, image deconvolution, spectral analysis, and broader statistical methodology. Bayesian software tools and practical workflows are also a recurring focus. Meetings are typically held about once per month and are announced via the Bayes Forum mailing list.


Abstract:

In 1968 Kolmogorov suggested basic information theory concepts "must and can be founded without recourse to the probability theory". In this talk I introduce a computable deterministic measure of information/entropy applicable to individual finite strings. With a series of Markov chain models, and known results from the study of non-linear dynamical systems and noisy chaos I demonstrate a precise correspondence with Shannon's probabilistic definition. Noting Shannon's entropy is but an average expectation for a source, we identify a significantly overlooked class of strings whose entropy is above the Shannon maximum average and challenging a popular conception, that of random strings having maximal entropy. We in fact observe information to be typically distributed unevenly implying lumpiness rather than homogeneity is the typical characteristic for entropy. For Shannon's units of information to be physically correct we recast his defining equation using a result from Speidel and Gulliver, namely: \[H_S= -e^{-\gamma$}\sum_i P_i\log(P_i)\] where $\gamma$ is the Euler-Mascheroni constant 0.57621... I consider briefly some implications for statistical mechanics and the second law.