Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Sample data first before load to numpy array #102

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

BirkhoffG
Copy link
Owner

Before, even if sample_frac is set, we use the entire data for building train_dataset, val_dataset, test_dataset.

When data is very large, this approach could unwantedly blow the memory because building datasets ( train_dataset etc.) assign new memory.

Now, I only load data with first sample_frac of data.

@BirkhoffG BirkhoffG added the bug Something isn't working label Jan 19, 2023
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

# for free to subscribe to this conversation on GitHub. Already have an account? #.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant