Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

load dataset cpu OOM #28

Open
MasaiahHan opened this issue Feb 25, 2025 · 3 comments
Open

load dataset cpu OOM #28

MasaiahHan opened this issue Feb 25, 2025 · 3 comments

Comments

@MasaiahHan
Copy link

Thanx for your Excellent work! When I try to load a JSONL file to finetune, I find out the CPU memory increasing and then out while loading the dataset. Looking for support.
@gaopengpjlab
Copy link
Contributor

gaopengpjlab commented Feb 25, 2025 •
edited
Loading

What's the scale of your dataset?

@MasaiahHan
Copy link
Author

What's the scale of your dataset?

The dataset consists of 1.4 Million images. I figure it out by reducing the global_bsz to avoid the problem. Thx for your kind reply! btw could you share a new wechat group QRcode? I think the qrcode in ReadME is out of date.

@ChinChyi
Copy link
Collaborator

@MasaiahHan We have updated the QR code. You can join the group to discuss the issues you encountered in more detail.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ChinChyi @gaopengpjlab @MasaiahHan