Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Running the model on TPUs? #55

Open
vessenes opened this issue Oct 25, 2019 · 2 comments
Open

Running the model on TPUs? #55

vessenes opened this issue Oct 25, 2019 · 2 comments

Comments

@vessenes
Copy link

Hi,

I have the 256 and 512 models working on GCP with a Tesla V100. Text generates, but slowly, and I'm wanting to get faster generation out of the system. I thought running CTRL on TPUs could get me faster text, but I have no idea how to do that.

Do you have an incantation or pointer that would let me point CTRL at a TPU?
@dimitri320
Copy link

Second this!

@keskarnitish
Copy link
Contributor

I haven't quite figured out how to get TPUs to be faster than GPUs for inference. I'll probably look into this soon. It's especially more complicated with top-k/nucleus sampling and other add-ons. Seems like others have found the same behavior.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vessenes @keskarnitish @dimitri320