conditional entropy computation

Description

condentropy takes two random vectors, X and Y, as input and returns the conditional entropy, H(X|Y), in nats (base e), according to the entropy estimator method. If Y is not supplied the function returns the entropy of X - see entropy.

Usage

condentropy(X, Y=NULL, method="emp")

Arguments

Details

• "emp" : This estimator computes the entropy of the empirical probability distribution.

• "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.

• "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.

• "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.

Value

condentropy returns the conditional entropy, H(X|Y), of X given Y in nats.

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

Cover, T. M. and Thomas, J. A. (1990). Elements of Information Theory. John Wiley, New York.

See Also

entropy, mutinformation, natstobits

Examples

  data(USArrests)
  dat<-discretize(USArrests)
  H <- condentropy(dat[,1], dat[,2], method = "mm")

conditional mutual information computation

Description

condinformation takes three random variables as input and computes the conditional mutual information in nats according to the entropy estimator method. If S is not supplied the function returns the mutual information between X and Y - see mutinformation

Usage

condinformation(X, Y, S=NULL, method="emp")

Arguments

Details

• "emp" : This estimator computes the entropy of the empirical probability distribution.

• "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.

• "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.

• "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.

Value

condinformation returns the conditional mutual information, I(X;Y|S), in nats.

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

Cover, T. M. and Thomas, J. A. (1990). Elements of Information Theory. John Wiley, New York.

See Also

mutinformation, multiinformation, interinformation, natstobits

Examples

  data(USArrests)
  dat<-discretize(USArrests)
  I <- condinformation(dat[,1],dat[,2],dat[,3],method="emp")

Unsupervized Data Discretization

Description

discretize discretizes data using the equal frequencies or equal width binning algorithm. "equalwidth" and "equalfreq" discretizes each random variable (each column) of the data into nbins. "globalequalwidth" discretizes the range of the random vector data into nbins.

Usage

discretize( X, disc="equalfreq", nbins=NROW(X)^(1/3) )

Arguments

Value

discretize returns the discretized dataset.

Author(s)

Patrick E. Meyer, Frederic Lafitte, Gianluca Bontempi, Korbinian Strimmer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

Dougherty, J., Kohavi, R., and Sahami, M. (1995). Supervised and unsupervised discretization of continuous features. In International Conference on Machine Learning.

Yang, Y. and Webb, G. I. (2003). Discretization for naive-bayes learning: managing discretization bias and variance. Technical Report 2003/131 School of Computer Science and Software Engineering, Monash University.

Examples

data(USArrests)
nbins<- sqrt(NROW(USArrests))
ew.data <- discretize(USArrests,"equalwidth", nbins)
ef.data <- discretize(USArrests,"equalfreq", nbins)
gew.data <- discretize(USArrests,"globalequalwidth", nbins)

entropy computation

Description

entropy takes the dataset as input and computes the entropy according to the entropy estimator method.

Usage

entropy(X, method="emp")

Arguments

Details

• "emp" : This estimator computes the entropy of the empirical probability distribution.

• "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.

• "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.

• "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.

Value

entropy returns the entropy of the data in nats.

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

J. Beirlant, E. J. Dudewica, L. Gyofi, and E. van der Meulen (1997). Nonparametric entropy estimation : An overview. Journal of Statistics.

Hausser J. (2006). Improving entropy estimation and the inference of genetic regulatory networks. Master thesis of the National Institute of Applied Sciences of Lyon.

See Also

condentropy, mutinformation, natstobits

Examples

  data(USArrests)
  H <- entropy(discretize(USArrests),method="shrink")

Information Theory package

Description

The package infotheo provide various estimators for computing information-theoretic measures from data

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

See Also

entropy, condentropy, mutinformation, condinformation, multiinformation, interinformation, natstobits

interaction information computation

Description

interinformation takes a dataset as input and computes the the interaction information among the random variables in the dataset using the entropy estimator method. This measure is also called synergy or complementarity.

Usage

interinformation(X, method="emp")

Arguments

Details

• "emp" : This estimator computes the entropy of the empirical probability distribution.

• "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.

• "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.

• "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.

Value

interinformation returns the interaction information (also called synergy or complementarity), in nats, among the random variables (columns of the data.frame).

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

Jakulin, A. and Bratko, I. (2004). Testing the significance of attribute interactions. In Proc. of 21st International Conference on Machine Learning (ICML).

McGill, W. J. (1954). Multivariate information transmission. Psychometrika, 19.

See Also

condinformation, multiinformation, mutinformation, natstobits

Examples

  data(USArrests)
  dat<-discretize(USArrests)
  ii <- interinformation(dat, method = "sg")

multiinformation computation

Description

multiinformation takes a dataset as input and computes the multiinformation (also called total correlation) among the random variables in the dataset. The value is returned in nats using the entropy estimator estimator.

Usage

multiinformation(X, method ="emp")

Arguments

Details

• "emp" : This estimator computes the entropy of the empirical probability distribution.

• "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.

• "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.

• "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.

Value

multiinformation returns the multiinformation (also called total correlation) among the variables in the dataset (in nats).

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

Studeny, M. and Vejnarova, J. (1998). The multiinformation function as a tool for measuring stochastic dependence. In Proceedings of the NATO Advanced Study Institute on Learning in graphical models,

See Also

condinformation, mutinformation, interinformation, natstobits

Examples

  data(USArrests)
  dat<-discretize(USArrests)
  M <- multiinformation(dat)

mutual information computation

Description

mutinformation takes two random variables as input and computes the mutual information in nats according to the entropy estimator method. If Y is not supplied and X is a matrix-like argument, the function returns a matrix of mutual information between all pairs of variables in the dataset X.

Usage

mutinformation(X, Y, method="emp")

Arguments

Details

• "emp" : This estimator computes the entropy of the empirical probability distribution.

• "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.

• "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.

• "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.

Value

mutinformation returns the mutual information I(X;Y) in nats.

Author(s)

Patrick E. Meyer

References

Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.

Cover, T. M. and Thomas, J. A. (1990). Elements of Information Theory. John Wiley, New York.

See Also

condinformation, multiinformation, interinformation, natstobits

Examples

  data(USArrests)
  dat<-discretize(USArrests)
  #computes the MIM (mutual information matrix)
  I <- mutinformation(dat,method= "emp")
  I2<- mutinformation(dat[,1],dat[,2])

convert nats into bits

Description

natstobits takes a value in nats (a double) as input and returns the value in bits (a double).

Usage

natstobits(H)

Arguments

Details

Information-theoretic quantities can have different units depending on the base of the logarithm used in their computation. All the function of tha package use a base e, hence the unit is the nat. The value in bit is given by using the base 2, hence the conversion is done by multiplying by log2(e) = 1.442695.

Value

natstobits returns a double that is the conversion of the nats value into bits.

Author(s)

Patrick E. Meyer

Examples

  data(USArrests)
  H <- entropy(discretize(USArrests))
  natstobits(H)