https://datascienceschool.net/view-notebook/ff9458c7156c4961b012c60e5fc97301/
================================================================================
* You almost can't perform linear regression analysis on following data
================================================================================
* When you perform "linear regression analysis" on "non-linear data"
you can use "linear basis function model" instead of using non-linear model
================================================================================
* General linear regression model
$$$y_i = \sum\limits_{i=1}^{D}w_ix_x = w^Tx$$$
* Number of dim of trainable param w = dim of feature vector
$$$x\in \mathbb{R}^{D} \Leftrightarrow w\in \mathbb{R}^{D}$$$
================================================================================
* Linear basis function models
* You don't directly perform linear combination on x
* You convert x via basis function to make new feature vector
* You perform linear regression model on new feature vector
* So, new dim of feature vector = dim of trainable param w
$$$\phi(\cdot): \mathbb{R}^{D} \rightarrow \mathbb{R}^{M}$$$, then, $$$w \in \mathbb{R}^{M}$$$
$$$y_i = \sum\limits_{j=1}^{M}w_j \phi_j(x) = w^T \phi(x)$$$
================================================================================
* Multinomial regression
* Multinomial regression has basis function as multinomial function
$$$1, x, x^2, \cdots, x^M$$$
* $$$y = w_0 + w_1x + w_2x^2+\cdots+w_Mx^M$$$
================================================================================
================================================================================
Overfitting
* When capacity of network is too big than dimension of feature vector
* When features in feature vector are not independet
In above situation, right answer can be multiple
================================================================================
Problem of overfitting
* With new feature data, it creates large error
* When train data comes, trainable parameters change largely
================================================================================
================================================================================