IEEE Transactions on Information Theory, 52(1):238–245, January 2006

• PDF

Abstract

A common approach for estimating a probability mass function when given a prior and moment constraints given by is to minimize the relative entropy between and subject to the set of linear constraints. In such cases, the solution is known to have exponential form. We consider the case in which the linear constraints are noisy, uncertain, infeasible, or otherwise “soft.” A solution can then be obtained by minimizing both the relative entropy and violation of the constraints . A penalty parameter weights the relative importance of these two objectives. We show that this penalty formulation also yields a solution with exponential form. If the distortion is based on an norm, then the exponential form of is shown to have exponential decay parameters that are bounded as a function of . We also state conditions under which the solution to the penalty formulation will result in zero distortion, so that the moment constraints hold exactly. These properties are useful in choosing penalty parameters, evaluating the impact of chosen penalty parameters, and proving properties about methods that use such penalty formulations. to maximizing entropy.

BibTeX

 @article{FrieGupt:2006,
  Author = {M. P. Friedlander and M. R. Gupta},
  Journal = {IEEE Transactions on Information Theory},
  Number = 1,
  Pages = {238-245},
  Title = {On minimizing distortion and relative entropy},
  Volume = 52,
  Year = 2006,
  Doi = {10.1109/TIT.2005.860448}
 }