M. Studeny and J. Vejnarova:
The multiinformation function as a tool for measuring stochastic dependence.
In Learning in Graphical Models (M. I. Jordan ed.)
Kluwer, Dordrecht 1998, pp. 261-298.
- Given a collection of random variables X(i) where
i belongs to a finite nonempty set N, the corresponding
multiinformation function ascribes (to every subset A of
N) the relative entropy of the joint distribution of X(A)
(that is, the random vector composed of the variables X(i) where
i belongs to A) with respect to the product of distributions
of individual random variables X(i) through i in A.
We argue that it is a useful tool for solving the problems concerning
stochastic (conditional) dependence and independence (at least in discrete
case). First, it makes possible to express the conditional mutual
information between X(A) and X(B) given X(C)
(for every disjoint subsets A,B,C of N) which can be
considered as a good measure of conditional stochastic depedence. Second,
one can introduce reasonable measures of dependence of level r
among variables of X(A) (where A is a subset of N
and 0 < r < card A) which are expressible by means of the
multiinformation function. Third, it enables one to derive theoretical
results on (nonexistence of an) axiomatic characterization of stochastic
conditional independence models.
In fact, the last part of the paper is nothing but a didactive proof
of the result from the paper:
- M. Studeny:
Conditional independence relations have no finite complete
characterization. In Information Theory, Statistical
Decision Functions and Random Processes. Transactions of the 11th
Prague Conference vol. B (S. Kubik, J.A. Visek eds.), Kluwer,
Dordrecht - Boston - London (also Academia, Prague) 1992, pp. 377-396.
- AMS classification 94A17, 68T30 (03B30, 62H05)
- conditional mutual information
- relative entropy
- level-specific measures of dependence
- (regular) inference rule
- perfect rule
pdf copy (converted postscript version) (366kB) is available.
The results of the paper has a connection to the following works:
- T.S. Han: Nonnegative entropy of multivariate symmetric
correlations. Information and Control 36
(1978), pp. 113-156.
- M. Studeny, P. Bocek:
CI-models arising among 4 random variables.
In Proceedings of the 3rd workshop WUPES, September 11-15, 1994,
Trest, Czech Republic, pp. 268-282.