Articles

A great article that first hits on the shortcomings of Pearson's Correlation before contrasting it with information theoretic methods of determining correlation.

Pearson's correlation attempts to uncover linear relationships between variables. In practice, very few relationships in the real world are linear, thus this method falls apart. Worse still, spurious correlations are common and very prevalent in scientific literature.

Mutual Information helps us uncover the dependence structure between two variables by comparing how much information is shared between them. Mutual Information considers the actual values presented rather than assuming a specific distribution like Pearson's Correlation does.