http://biz.chosun.com/site/data/html_dir/2019/02/08/2019020801834.html ================================================================================ AI language model - RNN - Order of input data is important - Information of first words generally disappear as train proceeds - So, RNN couldn't fully understand the sentence as the sentence becomes longer and longer - Attention-module based RNN - Attention-module dynamically pays more attention to important words regarless of the order of input data - Transformer - self-attention - Human doesn't intervene the process of attention - BERT - Use transformer in bidirectional way - doesn't use RNN structure (RNN is slow) - Performance: - Train "Lord of the ring" - Ask a question: where is the ring? - BERT can make a answer ================================================================================