feat: get vocab from tokenizer is better.#31

Open

FFengIll wants to merge 2 commits intoskeskinen:masterfrom FFengIll:feat/get_vocab

+29-20

Commits on Sep 14, 2023

feat: get vocab from tokenizer is better.
FFengIll
committed

Commits on Sep 19, 2023

feat: use vocab_size to loop vocab to avoid error.
FFengIll
committed