Fixes issues while loading word2vec and doc2vec models saved using old Gensim versions. Fix #2000, #1977
#2012
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Plan:
We need to cover as much as possible situations because this kind of problems are already starting to bother. |
menshikh-iv
reviewed
Apr 6, 2018
gensim/test/test_word2vec.py
Outdated
| saved_models_dir = datapath('old_w2v_models') | ||
| for old_version in old_versions: | ||
| model = word2vec.Word2Vec.load(os.path.join(saved_models_dir, 'w2v_{}.mdl'.format(old_version))) | ||
| self.assertTrue(len(model.wv.vocab) == 3) |
There was a problem hiding this comment.
add most_similar + update an model (similar for d2v)
piskvorky
requested changes
Apr 8, 2018
gensim/models/deprecated/doc2vec.py
Outdated
| new_model.docvecs.max_rawint = old_model.docvecs.__dict__.get('max_rawint') | ||
| new_model.docvecs.offset2doctag = old_model.docvecs.__dict__.get('offset2doctag') | ||
| else: | ||
| new_model.docvecs.max_rawint = len(old_model.docvecs.index2doctag) if old_model.docvecs.index2doctag else old_model.docvecs.count | ||
| new_model.docvecs.max_rawint = \ |
There was a problem hiding this comment.
Magic: definitely deserves a comment.
menshikh-iv
suggested changes
Apr 9, 2018
gensim/test/test_doc2vec.py
Outdated
| doc0_inferred = model.infer_vector(list(DocsLeeCorpus())[0].words) | ||
| sims_to_infer = model.docvecs.most_similar([doc0_inferred], topn=len(model.docvecs)) | ||
| self.assertTrue(sims_to_infer) | ||
|
|
There was a problem hiding this comment.
Can you add here save+load+infer_vector here (to be 100% sure that this persistent correctly)? Make sure that you used /tmp directory, check gensim.test.utils, you'll found needed functions (and same for w2v).
Also, please try to update model (as for w2v)
gensim/test/test_doc2vec.py
Outdated
| '3.0.0', '3.1.0', '3.2.0', '3.3.0', '3.4.0' | ||
| ] | ||
|
|
||
| saved_models_dir = datapath('old_d2v_models') |
There was a problem hiding this comment.
better datapath('old_d2v_models/d2v_{}.mdl') and format later
menshikh-iv
approved these changes
Apr 10, 2018
menshikh-iv
changed the title
word2vec and doc2vec models saved using old Gensim versions.word2vec and doc2vec models saved using old Gensim versions. Fix #2000, #1977
darindf
pushed a commit
to darindf/gensim
that referenced
this pull request
Apr 23, 2018
…g old Gensim versions. Fix piskvorky#2000, piskvorky#1977 (piskvorky#2012) * adds default values for attributes * ignore values for attributes that do not exist * adds unit test * fixes default values for missing attributes for older gensim models * adds unit test cases for loading really old gensim models * adds test cases for loading all old models * adds more tests post loading * handles loading d2v models saved using version <=0.12.2 * fix `max_rawint` value and PEP8 errors * adds saving and loading back tests * adds comments and fixes `max_rawint` * fix PEP8
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses #2000 and #1977. The issues were caused due to a few missing attributes (like
min_alpha_yet_reached,running_training_loss) in really old Gensim versions. I have added tests to loadword2vecmodel saved using Gensim version0.12.0anddoc2vecmodel saved using Gensim0.13.0. The tests also include checking online training and a similarity search post loading these old models.