Now, Pre-training of Language Model for Language Understanding is a significant step in the context of NLP.
A language model would be trained on a massive corpus, and then we can use it as a component in other models that need to handle language (e.g. using it for downstream tasks).
A Lanugage Model (LM) captures the distribution over all possible sentences.
- Input : a sentence
- Output : the probability of the input sentence
While language modeling is a typical unsupervised learning on massive corpus, we turn this into a sequence of supervised learning in this repo.
Autoregressive language model captures the distribution over the next token is based on all the previous token. In other words, it looks at the previous token, and predicts the next token.
The objective of Autoregressive language model is expressed in a formula as follows:
Because Autoregressive language model should be forward or backward, only one-way uni-directional context information can be used. Therefore, it's difficult to understand the context in both directions simultaneously.
RNNLM, ELMo are typical example of Autoregressive language model, and Unidirectional/Bidirectional LSTM language models are covered in this repo.
- cf. Bidirectional LSTM LM, ELMo use context in both directions. However, only shallow understanding is possible beacuase it use contexts that are independently learned in each direction.
- cf. For a detailed description of the model architecture, refer to the paper/repo in the Reference tab below.
Wikipedia regularly distributes the entire document. You can download Korean Wikipedia dump here (and English Wikipedia dump here).
Wikipedia recommends using pages-articles.xml.bz2, which includes only the latest version of the entire document, and is approximately 600 MB compressed (for English, pages-articles-multistream.xml.bz2).
You can use wikipedia_ko.sh script to download the dump on the latest Korean Wikipedia document. For English, use wikipedia_en.sh
example:
$ cd build_corpus
$ chmod 777 wikipedia_ko.sh
$ ./wikipedia_ko.sh
The downloaded dump using above shell script is in XML format, and we need to parse XML to text file. The Python script WikiExtractor.py in attardi/wikiextractor repo, extracts and cleans text from the dump.
example:
$ git clone https://github.com/attardi/wikiextractor
$ python wikiextractor/WikiExtractor.py kowiki-latest-pages-articles.xml
$ head -n 4 text/AA/wiki_02
<doc id="577" url="https://ko.wikipedia.org/wiki?curid=577" title="천문학">
천문학
천문학(天文學, )은 별이나 행성, 혜성, 은하와 같은 천체와, 지구 대기의 ..
</doc>
The extracted text is saved as text file of a certain size. To combine these, use build_corpus.py. The output corpus.txt contains 4,277,241 sentences, 55,568,030 words.
example:
$ python build_corpus.py > corpus.txt
$ wc corpus.txt
4277241 55568030 596460787 corpus.txt
Now, you need to split the corpus to train-set and test-set.
$ cat corpus.txt | shuf > corpus.shuf.txt
$ head -n 855448 corpus.shuf.txt > corpus.test.txt
$ tail -n 3421793 corpus.shuf.txt > corpus.train.txt
$ wc -l corpus.train.txt corpus.test.txt
3421793 corpus.train.txt
855448 corpus.test.txt
4277241 합계
Our corpus corpus.txt has 55,568,030 words, and 608,221 unique words. If the minimum frequency needed to include a token in the vocabulary is set to 3, the vocabulary contains 297,773 unique words.
Here we use the train corpus corpus.train.txt to build vocabulary.
The vocabulary built by train corpus contains 557,627 unique words, and 271,503 unique words that appear at least three times.
example:
$ python build_vocab.py --corpus build_corpus/corpus.train.txt --vocab vocab.train.pkl --min_freq 3 --lower
Namespace(bos_token='<bos>', corpus='build_corpus/corpus.train.txt', eos_token='<eos>', is_tokenized=False, lower=True, min_freq=3, pad_token='<pad>', tokenizer='mecab', unk_token='<unk>', vocab='vocab.train.pkl')
Vocabulary size: 271503
Vocabulary saved to vocab.train.pkl
Since the vocabulary file is too large(~1.3GB) to upload on this repo, I uploaded it to Google Drive.
vocab.train.pkl: [download]
$ python lm_trainer.py -h
usage: lm_trainer.py [-h] --train_corpus TRAIN_CORPUS --vocab VOCAB
--model_type MODEL_TYPE [--test_corpus TEST_CORPUS]
[--is_tokenized] [--tokenizer TOKENIZER]
[--max_seq_len MAX_SEQ_LEN] [--multi_gpu] [--cuda CUDA]
[--epochs EPOCHS] [--batch_size BATCH_SIZE]
[--clip_value CLIP_VALUE] [--shuffle SHUFFLE]
[--embedding_size EMBEDDING_SIZE]
[--hidden_size HIDDEN_SIZE] [--n_layers N_LAYERS]
[--dropout_p DROPOUT_P]
optional arguments:
-h, --help show this help message and exit
--train_corpus TRAIN_CORPUS
--vocab VOCAB
--model_type MODEL_TYPE
Model type selected in the list: LSTM, BiLSTM
--test_corpus TEST_CORPUS
--is_tokenized Whether the corpus is already tokenized
--tokenizer TOKENIZER
Tokenizer used for input corpus tokenization
--max_seq_len MAX_SEQ_LEN
The maximum total input sequence length after
tokenization
--multi_gpu Whether to training with multiple GPU
--cuda CUDA Whether CUDA is currently available
--epochs EPOCHS Total number of training epochs to perform
--batch_size BATCH_SIZE
Batch size for training
--clip_value CLIP_VALUE
Maximum allowed value of the gradients. The gradients
are clipped in the range
--shuffle SHUFFLE Whether to reshuffle at every epoch
--embedding_size EMBEDDING_SIZE
Word embedding vector dimension
--hidden_size HIDDEN_SIZE
Hidden size of LSTM
--n_layers N_LAYERS Number of layers in LSTM
--dropout_p DROPOUT_P
Dropout rate used for dropout layer in LSTM
example:
$ python lm_trainer.py --train_corpus build_corpus/corpus.train.txt --vocab vocab.train.pkl --model_type LSTM --batch_size 16
You can select your own parameter values via argument inputs.
Training a model with single GPU is not only very slow, it also limits adjusting batch size, model size, and so on.
To accelerate model training with multiple GPU and use large model, what you have to do is to include --multi_gpu flag like belows. For more details, please check here.
This example code trains unidirectional-LSTM model on the Wikipedia corpus using parallel training on 8 * V100 GPUs.
$ python lm_trainer.py --train_corpus build_corpus/corpus.train.txt --vocab vocab.train.pkl --model_type LSTM --multi_gpu
Namespace(batch_size=512, clip_value=10, cuda=True, dropout_p=0.2, embedding_size=256, epochs=10, hidden_size=1024, is_tokenized=False, max_seq_len=32, model_type='LSTM', multi_gpu=True, n_layers=3, shuffle=True, test_corpus=None, tokenizer='mecab', train_corpus='build_corpus/corpus.train.txt', vocab='vocab.train.pkl')
=========MODEL=========
DataParallelModel(
(module): LSTMLM(
(embedding): Embedding(271503, 256)
(lstm): LSTM(256, 1024, num_layers=3, batch_first=True, dropout=0.2)
(fc): Linear(in_features=1024, out_features=512, bias=True)
(fc2): Linear(in_features=512, out_features=271503, bias=True)
(softmax): LogSoftmax()
)
)
This example code trains Bidirectional-LSTM model on the Wikipedia corpus using parallel training on 8 * V100 GPUs.
$ python lm_trainer.py --train_corpus build_corpus/corpus.train.txt --vocab vocab.train.pkl --model_type BiLSTM --n_layers 1 --multi_gpu
Namespace(batch_size=512, clip_value=10, cuda=True, dropout_p=0.2, embedding_size=256, epochs=10, hidden_size=1024, is_tokenized=False, max_seq_len=32, model_type='BiLSTM', multi_gpu=True, n_layers=1, shuffle=True, test_corpus=None, tokenizer='mecab', train_corpus='build_corpus/corpus.train.txt', vocab='vocab.train.pkl')
=========MODEL=========
DataParallelModel(
(module): BiLSTMLM(
(embedding): Embedding(271503, 256)
(lstm): LSTM(256, 1024, batch_first=True, dropout=0.2, bidirectional=True)
(fc): Linear(in_features=2048, out_features=1024, bias=True)
(fc2): Linear(in_features=1024, out_features=512, bias=True)
(fc3): Linear(in_features=512, out_features=271503, bias=True)
(softmax): LogSoftmax()
)
)
A language model captures the distribution over all possible sentences. And, the best language model is one that the best predicts an unseen sentence. Perplexty is a very common measurement of how well a probability distribution predicts unseen sentences.
Perplexity : inverse probability of the given sentence, normalized by the number of words (by taking geometric mean)
As you can see from the above equation, perplexity is defined as the exponentiated negative average log-likelihood. In other words, maximizing probability is the same as minimizing perplexity.
And now, perplexity is the metric that we're going to be using. A low perplexity indicates that the probability distribution is good at predicting the sentence.
| Model | Loss | Perplexity |
|---|---|---|
| Unidirectional-LSTM | 3.496 | 33.037 |
| Bidirectional-LSTM | 1.896 | 6.669 |
| Bidirectional-LSTM-Large (hidden_size = 1024) | 1.771 | 5.887 |
- [Google DeepMind] WaveNet: A Generative Model for Raw Audio
- [Dan Jurafsky] CS 124: From Languages to Information at Stanford
- [attardi/wikiextractor] WikiExtractor
- [DSKSD] 6. Recurrent Neural Networks and Language Models
- [yunjey/pytorch-tutorial] Language Model (RNN-LM)
- [pytorch/examples] Word-level language modeling RNN
- [Mousa, Amr, and Björn Schuller] Contextual Bidirectional Long Short-Term Memory Recurrent Neural Network Language Models:A Generative Approach to Sentiment Analysis
- [Motoki Wu] The Bidirectional Language Model
- [matthew l]PyTorch Multi-GPU 제대로 학습하기
- [zhanghang1989/PyTorch-Encoding] PyTorch-Encoding , Issue: How to use the DataParallelCriterion, DataParallelModel