Limit the use of PreTrainedModel.device #16935
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sgugger
commented
Apr 25, 2022
Comment on lines
+542
to
+543
| if device is None: | ||
| device = self.device |
There was a problem hiding this comment.
Default to self.device here for a 100% backward compatible change.
|
The documentation is not available anymore as the PR was closed or merged. |
patrickvonplaten
approved these changes
Apr 25, 2022
elusenji
pushed a commit
to elusenji/transformers
that referenced
this pull request
Jun 12, 2022
* Limit the use of PreTrainedModel.device * Fix
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
I'm currently working on solutions to do model parallelism, offload weights to the CPU or the hard drive, and I've encountered some bugs linked to the way we use the
PreTrainedModel.device: it grabs the first parameter of the model to infer a device for the whole model. This doesn't work when the model is:So whenever it's possible, it would be great to rely on something else if we can, for instance some device where the inputs are. This PR does this for every use of this
deviceattribute in modeling_utils and generation_utils, with the exception of some code where there are no inputs passed so we generate them and have to use something for the device.If all works well, I plan to add all modeling files that make use of that attribute (when in the
dummy_inputs, I'll leave theself.devicebut outside of it, will grab the device of any inputs we have).