Machine Learning Theory

================================================================================

Short tutorials

KL divergence
  MinSeokHeo_KL_divergence
Cross Entropy
  MinSeokHeo_Cross_Entropy
Entropy, Shannon Entropy
  MinSeokHeo_Entropy_Shannon_entropy

================================================================================

JJ - Outline concept of a pattern recognition

http://www.kocw.net/home/search/kemView.do?kemId=1189957

01_002_Process_of_pattern_recognition
01_003_Feature_Feature_vector_Feature_space_Pattern_Scatter_plot

02_001_Introduce_vector_in_linear_algebra
02_002_Operation_on_vector
02_003_Orthogonal_projection
02_004_Linear_combination_Linearly_dependent_Linearly_independent
02_005_Basis_vector_Vector_space
02_006_Vector_space_Euclid_space_Distance_between_2_vectors_in_Euclidian_distance

03_001_Transpose_matrix_Squre_matrix_Diagonal_matrix_Scalar_matrix_Identity_matrix_Symmetry_matrix_Orthogonal_matrix
03_002_Trace_of_matrix_Determinant_value_of_matrix
03_003_Inverse_matrix_Positive-definite_Positive-semidefinite_matrix
03_004_Eigenvector_Eigenvalue
03_005_Linear_transform

005. Probability and statistics
  001_Population_Sample_Sampling_distribution_Mean_Variance_Covariance_Correlation_coefficient
  003
  004_Probability_space_Axioms_on_probability
  005_Marginal_probability_Conditional_probability_Joint_probability_Independent_trial_Law_of_the_total_probability
  006_Bayes_theorem

006. Random variable, Probability distribution
  001
  002
  003
  004

007. Bayesian Decision Theory
  001 Key points:
  - Discrete random variable
  - Cumulitive distribution function
  - Probability mass function
  - Probability density function
  - Univariate gaussian probability density function
  - Multivariate gaussian probability density function
  - Covariance
  002 Key points:
  - Normal distribution
  - Central limit theorem
  - The reason that gaussian distribution is much used
  - Relationship between pattern of covariance and data's distribution
  004 Key points:
  - Statistical methods
  - Likelihood ratio test
  - Maximum likelihood estimation
  - Likelihood
  005_Probability_of_error
  006_Bayes_risk
  007_Summary_3_variants_of_LRT_Bayes_criterion_MAP_criterion_ML_criterion

008. Bayesian Decision Theory with multiple classes
  001_Use_multi_classes_on_MAP_Bayes_criterion_considering_cost
  002_How_to_predict_likelihood_pdf_Parameter_estimation_Non_parameter_density_function_Maximum_likelihood_estimation
  003
  004
  005

009. Quadratic classifier
  001 Key points:
  - Various shapes of covariance matrix in Gaussian distribution function
  002 Key points:
  - Note
  003 Key points:
  - Note
  004 Key points:
  - Note
  005 Key points:
  - Note

010. Non-parametric density estimation
  001
  002
  10-04 Nonparametric density estimation
  10-05 Nonparametric density estimation
  006
  007

011. Clustering
  11-01 Clustring
  11-02 Clustring
  11-03 Clustring
  11-04 Clustring

012. Dimensionality reduction using PCA
  001
  002
  12-03 Short video 1 for the eigenvector and eigenvalue
  12-04 Short video 2 for the eigenvector and eigenvalue
  005
  12-06 Dimentionality reduction by PCA(principle component analysis)

013. Dimensionality reduction using LDA
  13-01 LDA(linear discriminant analysis)
  13-02 LDA(linear discriminant analysis)
  13-03 LDA(linear discriminant analysis)
  13-04 LDA(linear discriminant analysis)

================================================================================

TAcademy - Machine Learning Concepts

https://www.youtube.com/playlist?list=PL9mhQYIlKEheuxhyGbUIpKR1EFM-o0Br1

04. Decision Tree
05. Suppor Vector Machine(SVM)
10. Recurrent Neural Net(RNN)
12. Example applications with an applied deep learning

================================================================================

CWLee - Deep learning concept

https://www.youtube.com/playlist?list=PL1H8jIvbSo1q6PIzsWQeCLinUj_oPkLjc

02. Regression and gradient descent
03. Gradient Descent & Normal Equation
04. Logistic Regression
05. Loss function in logistic regression
06. Implementing a neural net(NN)
07. Backpropagation in neural net
08. softmax function
09. Convolutional Neural Net(CNN)
10. CNN Back Propagation
11. Local Minima
12. Unsupervised Pre-training for CNN
13. RNN Introduction
14. Back Propagation in RNN

================================================================================

HWLee - Outline concept of a random process

http://www.kocw.net/home/search/kemView.do?kemId=991018

01-01. Probability axioms and random variables
01-02. Probability axioms and random variables
02-01. Function of random variables, Definitions of convergence, Convergence in probability, Convergence with probability 1, Convergence in distribution
02-02. Function of random variables, Definitions of convergence, Convergence in probability, Convergence with probability 1, Convergence in distribution
03-01. Useful inequalities and law of large numbers. Central limit theorem. Markov inequality. Chebyshev inequality. Chernoff bound.
03-01. Useful inequalities and law of large numbers. Central limit theorem. Markov inequality. Chebyshev inequality. Chernoff bound.
04-01. Bernoulli process and Poisson process Definitions and properties of Bernoulli and Poisson processes
04-01. Bernoulli process and Poisson process Definitions and properties of Bernoulli and Poisson processes
05-01. Discrete-time Markov chains and steady-state behavior Definition, state transition probability, Markov property
06-01. Mixing time and midterm review Role of second largest eigenvalues and midterm review
06-01. Mixing time and midterm review Role of second largest eigenvalues and midterm review
07-01. M/M/1 queues Poisson arrival and exponential service, analysis of waiting times
07-01. M/M/1 queues Poisson arrival and exponential service, analysis of waiting times
08-01. M/G/1 queues and Pollaczek- Khinchin formula Definition of M/G/1 queue and derivation of Pollaczek-Khinchin formula
08-01. M/G/1 queues and Pollaczek- Khinchin formula Definition of M/G/1 queue and derivation of Pollaczek-Khinchin formula
09-01. Estimation theory and Expectation- Maximization (EM) algorithm Bayesian estimation, expectation maximization
09-01. Estimation theory and Expectation- Maximization (EM) algorithm Bayesian estimation, expectation maximization
10-01. Hidden Markov models (HMM) Modeling uncertain pheonomena using hidden Markov models
10-02. Hidden Markov models (HMM) Modeling uncertain pheonomena using hidden Markov models
11-01. Counting processes and Renewal processes Definition of counting and renewal processes, and analysis
11-01. Counting processes and Renewal processes Definition of counting and renewal processes, and analysis
12-01. Randomized algorithms Applications of probability and stochastic processes to randomized algorithms
12-01. Randomized algorithms Applications of probability and stochastic processes to randomized algorithms
13-01. Randomized algorithms and course review Applications of probability and stochastic processes to randomized algorithms
13-01. Randomized algorithms and course review Applications of probability and stochastic processes to randomized algorithms

================================================================================

DHKim - Data Science

https://datascienceschool.net

2. Math for data science
(03) Advanced linear algebra
  001_Basic_of_linear_algebra_and_analytic_geometry
  002_Coordinates_and_transformation
(04) Optimization using SciPy
  000_Basic_of_optimization
  001_Optimization_theory
  002_Optimization_with_constraints
  003_Linear_programming_Quadratic_programming
(05) Probability theory
  001_Set_theory
  002_Mathematical_definition_of_probability_and_its_meaning
  003_Probability_distribution_function
(06) Probability distribution
  06.06_Bernoulli_distribution
  07.07_Dirichlet_distribution
(07) Correlation relationship
  001_Multivariate_discrete_random_variable
  002_Multivariate_continuous_random_variable
  003_Independant_and_correlation_of_random_variables
  004_Covariance_and_correlation_coefficient
(09) Test and inference
  09.01_Inference_parameter
  005_MLE_mechanism

3. Regression ananalysis and Timeseries analysis
(01) Introduction to package and dataset for regression analysis
  01.01_statsmodels_package
  01.02_scikit-learn_package

(02) Basic of linear regression
  02.01_Basic_of_linear_regression
  02.03_Range_datatype_independent_variable
  02.05_Geometric_perspective_on_linear_regression_analysis

(05) Basis function and Normalization
  05.01_Multinomial_regression_Overfitting
  05.03_Multicollinearity

(07) ARIMA time series model
  07.04_ARMA_model

4. Machine learning for data science
(09) Combining model (Ensemble)
  09.01_Ensemble

(12) Optimize model and Practical classification
  12.02_Imbalanaced_data

(15) Probabilistic graph model
  14.01_Graph_theory
  14.02_Probabilistic_graph_model
  14.03_Network_inference
(16) State space model
  16.01_Hidden_markov_model
(17) Monte Carlo
  001_Monte_Carlo_Bayesian_analysis
(18) Mixture model and Variational inference
  18.01_Gaussian_mixtuure_model_and_EM_algorithm

5. Deep learning for data science
(06) RNN
  06.01_Word_embedding_Word2vec

008-001. introduction to pandas
008-002. ID data, load csv, create csv, export csv
008-003. dataframe indexer, loc[], iloc[], at[], iat[]
008-004. dataframe manipulating data
010-001. nltk package for natural language processing
025-001. bernoulli distribution
031-001. meaning of test and parameter estimation
031-002. testing and p-value
045-001. K-Means clustring
048-002. naive bayes classification model

================================================================================

Terry - Deep Learning Topics

https://www.youtube.com/playlist?list=PL0oFI08O71gKEXITQ7OG2SCCXkrtid7Fq

001_005_Metrics_for_deep_learrning_classification_Accuracy_Precision_Recall
001_006_ROC_curve_AUC_Precision_Recall

================================================================================

icmoon - Machine Learning Basic

https://www.youtube.com/playlist?list=PLbhbGI_ppZISMV4tAWHlytBqNq1-lb8bz

001. Week 01. Motivations and Basics - 01. Motivation
002. Week 01. Motivations and Basics - 02. MLE(maximum likelihood estimation)
003. Week 01. Motivations and Basics - 03. MAP(maximum posteriori estimation)
004. Week 01. Motivations and Basics - 04. Probability and Distribution
005. Week 02. Fundamentals of Machine Learning - 01. Rule-Based machine learning
008. Week 02. Fundamentals of Machine Learning - 04. Entropy and Info. Gain
009. Week 02. Fundamentals of Machine Learning - 05. How to create a Disicion tree
010. Week 03. Naive Bayes Classifier - 01. Optimal Classification
011. Week 03. Naive Bayes Classifier - 02. Conditional Independence
018. Week 04. Logistic Regression - 05. How Gradient method works
061. Week 10. Sampling Based Inference - 01. Forward Sampling
2 Fundamentals of Machine Learning | Lecture 2 Intro. to Rule-Based
2 Fundamentals of Machine Learning | Lecture 3 Introduction to Decision Tree
3 Naive Bayes Classifier | Lecture 3 Naive Bayes Classifier

3 Naive Bayes Classifier | Lecture 4 Naive Bayes Classifier with Matlab

4 Logistic Regression | Lecture 1 Decision Boundary

4 Logistic Regression | Lecture 2 Introduction to Logistic Regression

4 Logistic Regression | Lecture 3 Logistic Regression Parameter Approximation 1

4 Logistic Regression | Lecture 4 Gradient method

4 Logistic Regression | Lecture 5 How Gradient method works

4 Logistic Regression | Lecture 6 Logistic Regression Parameter Approximation 2

4 Logistic Regression | Lecture 7 Naive Bayes to Logistic Regression

4 Logistic Regression | Lecture 8 Naive Bayes vs Logistic Regression

5 Support Vector Machine | Lecture 1 Decision boundary with Margin

5 Support Vector Machine | Lecture 2 Maximizing the Margin

5 Support Vector Machine | Lecture 3 SVM with Matlab

5 Support Vector Machine | Lecture 4 Error Handling in SVM

5 Support Vector Machine | Lecture 5 Soft Margin with SVM

5 Support Vector Machine | Lecture 6 Rethinking of SVM

5 Support Vector Machine | Lecture 7 Primal, Dual with KKT Condition

5 Support Vector Machine | Lecture 8 Kernel

5 Support Vector Machine | Lecture 9 SVM with Kernel

032. Week 06. Training, Testing, Regularization - 01. Overfitting, Underfitting
033. Week 06. Training, Testing, Regularization - 02. Trade-off relation between bias and variance

6 Training Testing and Regularization | Lecture 3 Occam’s razor

6 Training Testing and Regularization | Lecture 4 Cross Validation

6 Training Testing and Regularization | Lecture 5 Performance Metrics

6 Training Testing and Regularization | Lecture 6 Regularization

6 Training Testing and Regularization | Lecture 7 Regularization Approximation

7 Bayesian Network | Lecture 1 Probability Concepts

7 Bayesian Network | Lecture 2 Probability Theorems

7 Bayesian Network | Lecture 3 Interpretation of Bayesian Network

7 Bayesian Network | Lecture 4 Bayes Ball Algorithm

7 Bayesian Network | Lecture 5 Factorization of Bayesian networks

7 Bayesian Network | Lecture 6 Inference Question on B. Networks

7 Bayesian Network | Lecture 7 Variable Elimination

7 Bayesian Network | Lecture 8 Potential Function and Clique Graph

7 Bayesian Network | Lecture 9 Potential Function and Clique Graph

008 K-Means clustering and Gaussian mixture model
  001_K-Means_Algorithm
  002_K-Means_Algorithm
  003_Multinomial_distribution

8 K-Means Clustering and Gaussian Mixture Model | Lecture 4 Multivar.

8 K-Means Clustering and Gaussian Mixture Model | Lecture 5 G.M.M

8 K-Means Clustering and Gaussian Mixture Model | Lecture 6 EM(Elimination-Maximization) step

8 K-Means Clustering and Gaussian Mixture Model | Lecture 7 Relation

8 K-Means Clustering and Gaussian Mixture Model | Lecture 8 EM(Elimination-Maximization)

8 K-Means Clustering and Gaussian Mixture Model | Lecture 9 Deriv. EM

009 Hidden markov model
  001_Concept_of_hidden_markov_model
  002_Joint_probability_Marginal_probability_of_HMM
  003_Recursion_and_Dynamic_programming_on_repeating_problem_Forward_probability_Backward_probability_in_HMM
  004_Viterbi_Decoding_Algorithm
  005_Baum-welch_algorithm
010 Sampling Based Inference
  10 Sampling Based Inference | Lecture 1 Forward Sampling
  002_Rejection_Sampling
  10 Sampling Based Inference | Lecture 3 Importance Sampling
  10 Sampling Based Inference | Lecture 4 Markov Chain
  10 Sampling Based Inference | Lecture 5 Markov Chain for Sampling
  10 Sampling Based Inference | Lecture 6 Metropolis-Hastings Algorithm
  007_Gibbs_Sampling
  008_Understand_LDA
  009_Gibbs_Sampling_for_LDA_1
  10 Sampling Based Inference | Lecture 10 Gibbs Sampling for LDA (2)

================================================================================
Todo

ICMoon - Machine Learning Advanced

Genetic algorithm

Genetic_algorithm_GRIT_A

================================================================================
Local storage