Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Using MultiDataset and ResizedDataset simultaneously gives rise to an AttributeError #9

Closed
sangho-vision opened this issue Sep 1, 2020 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@sangho-vision
Copy link

Hello,
when I used ResizedDataset and MultiDataset, I encountered the following error:

# Used code
import webdataset as wds
dataset = wds.Dataset(url).shuffle(100).decode()
dataset = wds.ResizedDataset(dataset, 5)
loader = wds.MultiDataset(dataset, workers=2, nominal=5, pin_memory=False).shuffle(2).batched(2)
for sample in loader:
   ...
Process Process-1:
Traceback (most recent call last):
...
  File ".../webdataset/multi.py", line 64, in _parallel_job
    D("job", i, "done", dataset.sample_urls)
AttributeError: 'ResizedDataset' object has no attribute 'sample_urls'
Process Process-2:
Traceback (most recent call last):
...
  File ".../webdataset/multi.py", line 64, in _parallel_job
    D("job", i, "done", dataset.sample_urls)
AttributeError: 'ResizedDataset' object has no attribute 'sample_urls'

Please check out this issue.
Thank you.

@tmbdev tmbdev self-assigned this Sep 2, 2020
@tmbdev tmbdev added the bug Something isn't working label Sep 2, 2020
@tmbdev
Copy link
Collaborator

tmbdev commented Sep 2, 2020

ResizedDataset was primarily intended as a workaround for limitations of DataLoader with IterableDatasets, so you may not need it with MultiDataset.

However, this still looks like a bug and I'll look into fixing it.

@tmbdev
Copy link
Collaborator

tmbdev commented Sep 17, 2020

It's fixed. You can use ResizedDataset with MultiDataset now.

@tmbdev tmbdev closed this as completed Sep 17, 2020
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants