INFORMATION AND ITS METRIC “When one speaks of information one refers to that entity shared by all sources that are equivalent up to recoding. This observation suggests a definition of information. Information of the source S is the equivalence class of all recodings of the symbol sequences from S. This is noteworthy since information generally is left undefined in information theory. Information theory only considers the amount of information or rates of production, loss, and transmission, as measured by various entropies. [...]
Recoding R partitions the space I of unique information sources into mutually disjoint subsets. [...]
This suggests the definition of the more abstract information space I [...] as the set of equivalence classes of I under recodings R. [...] Elements of shall be denoted X, Y, and so on. [...] The elements of I shall be the objects of interest in the following. The necessary logical distinction between the class X and its constituent sources will be blurred in the following. A reference to X as an information source should be construed as connoting the common properties of its members. As a source, X is the generic source. We can speak, in a similar vein, of the events or measurements of source X. [...]
Theorem: d is a metric and (I, d) is a metric space. [...]
The theorem indicates that the space of information sources has quite a bit of topological structure. For example, the notion of [epsilon]-balls of “close” information sources, the continuity of functions on information sources, and the limits and convergence of sequences of information sources, can be developed. These and numerical computations of information distances will follow in a sequel. [...]
CONCLUDING REMARKS
One question that arises in this development is why not simply use mutual information instead of the information metric. Aside from the pseudo-geometric picture we have presented, we note that the former measures only a kind of informational correlation. The information metric, however, quantifies the degree of recoding equivalence. And so, it provides some insight into the nature of information itself. Mutual information is a derivative concept that simply reflects the properties Shannon entropy and no more.
The foregoing mathematical development instantiates a particular philosophical view- point, that of phenomenology. All that an observer has to work with in developing an understanding of the world are finite measurements and the attendant information. This intrinsic finiteness derives first and foremost from the limited computation resources available to an observer in a finite space-time region. The information space, as developed here, is the substrate for all perception, quantification, and modeling building. This is then structured with the pseudo-geometry as we have just shown. Only under suitable restrictions is one justified in using observations to form probabilities via (say) frequencies of events.
Information theory was founded on a quantitative measure of the amount of information. The foregoing has given a formal definition of information itself in terms of the equivalence class structure of sources. But what of the “meaning” of this information? A motivation of this work, unstated until this point, was the conviction that an understanding of the topological structure of the metric lattice of inferential logic is necessary for developing a quantitative measure of meaning and of context. Thus, we offer no immediate answer to the question, only the hope that progress can be made. We shall return to this question in the future.”
@inproceedings{crutchfield1990information,
title={Information and Its Metric},
author={Crutchfield, JP},
booktitle={Nonlinear structures in physical systems: pattern formation, chaos, and waves: proceedings of the Second Woodward Conference, San Jose State University, November 17-18, 1989},
pages={119},
year={1990},
organization={Springer}
}
Download: http://scholar.google.com/scholar?cluster=2410770660010935934
admin is discussing. 
http://www.philosophybro.com/2011/04/mailbag-monday-what-do-we-know.html
http://en.wikipedia.org/wiki/Gettier_problem
http://www.iep.utm.edu/gettier/