New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

samples_per_second_per_gpu or tokens_per_second_per_gpu? #262

Open

Muennighoff opened this issue Apr 29, 2024 · 1 comment

Contributor

Muennighoff commented Apr 29, 2024

I'm probably missing something but isn't this tokens per second per GPU:

open_lm/open_lm/train.py

Line 282 in 083fa31

samples_per_second_per_gpu = inputs.numel() / batch_time_m.val

inputs.numel() gives all tokens; for samples it would be inputs.shape[0], no?

Collaborator

achalddave commented Apr 29, 2024 via email

Yeah it’s poorly named; it is in fact tokens per second. Happy to accept a PR for a fix!

On Mon, Apr 29, 2024 at 1:29 PM Niklas Muennighoff ***@***.***> wrote: I'm probably missing something but isn't this tokens per second per GPU: https://github.com/mlfoundations/open_lm/blob/083fa31449c3456e889269e44913578acfced67a/open_lm/train.py#L282 inputs.numel() gives all tokens; for samples it would be inputs.shape[0], no? — Reply to this email directly, view it on GitHub <#262>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI5L4NIJOCZJVC74SMAD5TY72URNAVCNFSM6AAAAABG7AA4DCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DSOJRGUYDKMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

# for free to join this conversation on GitHub. Already have an account? # to comment