https://datascienceschool.net/view-notebook/ff9458c7156c4961b012c60e5fc97301/ ================================================================================ * You almost can't perform linear regression analysis on following data ================================================================================ * When you perform "linear regression analysis" on "non-linear data" you can use "linear basis function model" instead of using non-linear model ================================================================================ * General linear regression model $$$y_i = \sum\limits_{i=1}^{D}w_ix_x = w^Tx$$$ * Number of dim of trainable param w = dim of feature vector $$$x\in \mathbb{R}^{D} \Leftrightarrow w\in \mathbb{R}^{D}$$$ ================================================================================ * Linear basis function models * You don't directly perform linear combination on x * You convert x via basis function to make new feature vector * You perform linear regression model on new feature vector * So, new dim of feature vector = dim of trainable param w $$$\phi(\cdot): \mathbb{R}^{D} \rightarrow \mathbb{R}^{M}$$$, then, $$$w \in \mathbb{R}^{M}$$$ $$$y_i = \sum\limits_{j=1}^{M}w_j \phi_j(x) = w^T \phi(x)$$$ ================================================================================ * Multinomial regression * Multinomial regression has basis function as multinomial function $$$1, x, x^2, \cdots, x^M$$$ * $$$y = w_0 + w_1x + w_2x^2+\cdots+w_Mx^M$$$ ================================================================================ ================================================================================ Overfitting * When capacity of network is too big than dimension of feature vector * When features in feature vector are not independet In above situation, right answer can be multiple ================================================================================ Problem of overfitting * With new feature data, it creates large error * When train data comes, trainable parameters change largely ================================================================================ ================================================================================