This is personal study note
Copyright and original reference:
https://www.youtube.com/watch?v=R7GyEGsQ_ys&list=PLsri7w6p16vscJ4rkstBZQJqNtZf8Tkxq&index=2
================================================================================
The smaller residual $$$\epsilon$$$ becomes, it's better for the regression model to explain the pattern between X and Y
================================================================================
Regression equation from population: $$$Y_i = \beta_0 + \beta_i X_i$$$
Regression equation from sample: $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_i X_i$$$
Good regreesion analysis:
$$$\hat{\beta}$$$ should be near to $$$\beta$$$
But note that $$$\hat{\beta}$$$ and $$$\beta$$$ don't need to be same
================================================================================
But to make them almost same, you can use residual $$$\epsilon$$$
$$$Y_i = \beta_0 + \beta_i X_i$$$
$$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_i X_i + \hat{\epsilon}_i$$$
If $$$\hat{\epsilon}_i$$$ can be minized, $$$Y_i = \beta_0 + \beta_i X_i$$$ and $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_i X_i + \hat{\epsilon}_i$$$ become almost same
================================================================================
Method of least squares:
- The method which minimizes $$$\sum\limits \hat{\epsilon}^2$$$
Sum of "squared $$$\epsilon$$$" becomes minimum, regression model $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i + \hat{\epsilon}_i$$$ is the best
================================================================================
Maximum likelihood method
- Unlike "method of least squares" where summed cost should be minimized
- The goal of "maximum likelihood method" is to maximize likelihood of explaining parameters of population
- Probability about the likelihood is continuously changing
- Likelihood continusouly chases the population's regression equation
================================================================================
Result from "method of least squares" and "maximum likelihood method" is ultimately same
================================================================================
Gauss markov theorem
When following regression equation is given, $$$Y = \beta_0 + \beta_1 X_1$$$
When you estimate above regression equation, using "method of least squares" is the best is proved by Gauss and markov
Gauss-markov theorem:
- Under following conditions
1. independent variables are not random
2. Expectation value about the residual is 0, $$$E(\epsilon)=0$$$
3. $$$E(\epsilon_i,\epsilon_j)=0$$$, $$$E(\epsilon_i,\epsilon_j)=\gamma^2$$$
- using "method of least squares" is BLEU (Best, Linear model, Estimation, Unbiased)