Kullback–Leibler divergence

In mathematical statistics, the Kullback–Leibler (KL) divergence (also called relative entropy and I-divergence), denoted D KL ( P ∥ Q ) {\displaystyle D_{\text{KL}}(P\parallel Q)} , is a type of statistical distance: a measure of how much an approximating probability distribution Q is different from a true probability distribution P. Mathematically, it is defined as D KL ( P ∥ Q ) = ∑ x ∈ X P ( x ) log ⁡ P ( x ) Q ( x ) . {\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\,\log {\frac {P(x)}{Q(x)}}{\text{.}}} A simple interpretation of the KL divergence of P from Q is the expected excess surprisal from using the approximation Q instead of P when the actual is P. While it is a measure of how different two distributions are and is thus a distance in some sense, it is not actually a metric, which is the most familiar and formal type of distance.

Source: Wikipedia — Kullback–Leibler divergence (CC BY-SA 4.0)