We know that the empirical risk is :
$L_s = frac{1}{n} sum_{i=1}^{n} l(f(x_i),z_i)$
where, $n$ = number of samples,$l(f(x),z)$ is a loss function, $S = (z_1,…,z_n)$ are the provided samples to testtrain on, $f(x_i)$ is the output of a learning algorithm and $z_i in Z $ is $(x_i, y_i)$
Some current literature like this (page 3 section 3.2), indicate that the true risk to be the expected value with respect to joint distribution of $n$ samples.
$L_mu = E_{Z^n}[L_s]$ $_{(1)}$
I do not understand how can the true risk be just the expected value of the empirical risk
This is how far I got :
We know that true risk is defined as :
$L_u = int l(f(x),z) dP(x,y) $
Now to write it in a summation form I can assume $n$ to be arbitrarily large, $L_u = frac{1}{n} sum_{i=1}^{n} l(f(x_i),z_i).P(x_i,y_i)$ $_{(2)}$
Now have no idea about the next step to reconcile equation $(2)$ and $(1)$
Best Answer
You don't need to assume $n$ to be arbitrarily large. It's just linearity of expectation:$DeclareMathOperator{E}{mathbb E}$ begin{align} E[text{empirical risk}] &= Eleft[ frac1n sum_{i=1}^n l(f(x_i), z_i) right] \&= frac1n sum_{i=1}^n Eleft[ l(f(x_i), z_i) right] \&= frac1n sum_{i=1}^n int l(f(x_i), z_i) ,mathrm{d}P(x_i, z_i) \&= frac1n sum_{i=1}^n text{true risk} \&= text{true risk} .end{align}
Similar Posts:
- Solved – How to use the standard regression assumptions to prove that $hat{sigma}^2$ is an unbiased estimator of $sigma^2$
- Solved – Expected Value of Maximum likelihood mean for Gaussian Distribution
- Solved – Expected Value of Maximum likelihood mean for Gaussian Distribution
- Solved – How to show that the average empirical risk is equal to the true risk for a binary classifier
- Solved – Find the UMVUE of $frac{mu^2}{sigma}$ where $X_isimmathsf N(mu,sigma^2)$