This is personal study note Copyright and original reference: https://www.youtube.com/watch?v=R7GyEGsQ_ys&list=PLsri7w6p16vscJ4rkstBZQJqNtZf8Tkxq&index=2 ================================================================================ The smaller residual $$$\epsilon$$$ becomes, it's better for the regression model to explain the pattern between X and Y ================================================================================ Regression equation from population: $$$Y_i = \beta_0 + \beta_i X_i$$$ Regression equation from sample: $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_i X_i$$$ Good regreesion analysis: $$$\hat{\beta}$$$ should be near to $$$\beta$$$ But note that $$$\hat{\beta}$$$ and $$$\beta$$$ don't need to be same ================================================================================ But to make them almost same, you can use residual $$$\epsilon$$$ $$$Y_i = \beta_0 + \beta_i X_i$$$ $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_i X_i + \hat{\epsilon}_i$$$ If $$$\hat{\epsilon}_i$$$ can be minized, $$$Y_i = \beta_0 + \beta_i X_i$$$ and $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_i X_i + \hat{\epsilon}_i$$$ become almost same ================================================================================ Method of least squares: - The method which minimizes $$$\sum\limits \hat{\epsilon}^2$$$ Sum of "squared $$$\epsilon$$$" becomes minimum, regression model $$$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i + \hat{\epsilon}_i$$$ is the best ================================================================================ Maximum likelihood method - Unlike "method of least squares" where summed cost should be minimized - The goal of "maximum likelihood method" is to maximize likelihood of explaining parameters of population - Probability about the likelihood is continuously changing - Likelihood continusouly chases the population's regression equation ================================================================================ Result from "method of least squares" and "maximum likelihood method" is ultimately same ================================================================================ Gauss markov theorem When following regression equation is given, $$$Y = \beta_0 + \beta_1 X_1$$$ When you estimate above regression equation, using "method of least squares" is the best is proved by Gauss and markov Gauss-markov theorem: - Under following conditions 1. independent variables are not random 2. Expectation value about the residual is 0, $$$E(\epsilon)=0$$$ 3. $$$E(\epsilon_i,\epsilon_j)=0$$$, $$$E(\epsilon_i,\epsilon_j)=\gamma^2$$$ - using "method of least squares" is BLEU (Best, Linear model, Estimation, Unbiased)