Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

batch config issue? #17

Closed
samyam opened this issue Feb 5, 2020 · 2 comments · Fixed by #33
Closed

batch config issue? #17

samyam opened this issue Feb 5, 2020 · 2 comments · Fixed by #33
Assignees
@jeffra @samyam @tjruwase
Labels
bug Something isn't working enhancement New feature or request

Comments

@samyam
Copy link
Contributor

samyam commented Feb 5, 2020

There are a few things in configure train batch size that does not seem correct to me, and there are few things that we do not currently support.

  1. The following assertion

train_batch_size == train_micro_batch_size_per_gpu * gradient_accumulation_step * world_size

should always hold but currently it does not in some cases.
For example, when train_micro_batch_size_per_gpu and gradient accumulation steps are None in the ds_cofig its initialized to train_batch_size and 1 respectively which leads to

train_batch_size == train_batch_size * 1 * world_size

  1. if train_micro_batch_size_per_gpu > per_device_batch_size, we should throw a config error. Currently, its assigned to be equal to per_device_batch_size.

  2. We do not currently support the user providing only the train_micro_batch_size or train_micro_batch_size and gradient _accumulation_steps.
@samyam samyam added bug Something isn't working enhancement New feature or request invalid labels Feb 5, 2020
@samyam samyam linked a pull request Feb 7, 2020 that will close this issue
Samyamr/batchconfig #33
Merged
@samyam
Copy link
Contributor Author

samyam commented Feb 7, 2020

Fixed with pull request Samyamr/batchconfig #33

@samyam samyam closed this as completed Feb 7, 2020
jeffra pushed a commit to jeffra/DeepSpeed that referenced this issue May 15, 2020
@samyam
Merge pull request deepspeedai#17 from microsoft/samyamr/deepspeed_ac…
925cfdc 
…tivation_checkpointing

Adding activation checkpointing as deepspeed file
@gongwei-130 gongwei-130 mentioned this issue Aug 7, 2020
Open
@TonyTangYu TonyTangYu mentioned this issue Aug 21, 2020
Open
@GrvLeo GrvLeo mentioned this issue Oct 22, 2020
Closed
rraminen pushed a commit to rraminen/DeepSpeed that referenced this issue Jun 4, 2021
@jithunnair-amd
Merge pull request deepspeedai#17 from rraminen/IFU_5_27
1850f88 
IFU-master-2021-05-27
@garvct garvct mentioned this issue Jun 29, 2021
Open
pengwa pushed a commit to pengwa/DeepSpeed that referenced this issue Oct 14, 2022
@conglongli
Curriculum learning support (deepspeedai#17)
db97cd2 
* initial commit

* script fix
@lambda7xx lambda7xx mentioned this issue Feb 24, 2023
Closed
@rafael-ariascalles
Copy link

I am having this same issue using 0.9 but not 0.8 (using a was p4 machine)

@phalexo phalexo mentioned this issue Oct 11, 2023
Open
baodii pushed a commit to baodii/DeepSpeed that referenced this issue Oct 17, 2023
@Liangliang-Ma
add aio in xpu_upstream (deepspeedai#17)
32ecdae
radna0 pushed a commit to radna0/DeepSpeed-XLA that referenced this issue Feb 5, 2025
@dienachtderwelt
Merge pull request deepspeedai#17 from ykdai:dev
23fcea7 
Add light model
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@jeffra jeffra

@samyam samyam

@tjruwase tjruwase

Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Samyamr/batchconfig deepspeedai/DeepSpeed
4 participants
@jeffra @samyam @tjruwase @rafael-ariascalles