Calculus
Linear Algebra
- scalar
- vector
- row vector
- column vector
- matrix
Probability and Statistics
Information theory
KL-divergence(or relative entropy)
Kullback-Leibler divergence, a.k.a. relative entropy
For discrete probability distribution $P$ and $Q$, the Kullback-Leibler divergence from $Q$ to $p$ is defined to be
For distributions $P$ and $Q$ of a continuous random variable, the Kullback-Leibler divergence is defined to be the integral:
The above form of KL-divergence can also be called as entropy of $P$ relative to $Q$.
Cross entropy
The cross entropy for distribution $p$ and $q$ over a given set is defined as follows
Note that often we denote $q$ as the true distribution and $p$ as an “unnatural” probability distribution.