Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

fairseq/examples/roberta/README.pretraining.md #1073

Closed
yucoian opened this issue Aug 27, 2019 · 2 comments
Closed

fairseq/examples/roberta/README.pretraining.md #1073

yucoian opened this issue Aug 27, 2019 · 2 comments

Comments

@yucoian
Copy link

yucoian commented Aug 27, 2019

When I followed this tutorial for pretraining the Roberta on a 16GPUs machine, I found it didn't output a step result, like a loss, ppl etc. However, the GPUs and CPUs utilization both are 100%. Is there any solutions?
@ngoyal2707
Copy link
Contributor

Can you please share the logs?
GPUs being stuck exactly at 100% and not doing anything could mean different things, it could mean one or more GPU oomed in distributed training.

@lematt1991
Copy link
Contributor

Closing due to inactivity. Please open a new issue if you are still having problems.

facebook-github-bot pushed a commit that referenced this issue Mar 8, 2020
@facebook-github-bot
Allow dictionaries to overwrite entries with #fairseq:overwrite comme…
937535d 
…nt (#1073)

Summary:
[This commit](dd1298e) made it so that duplicate entries in a dictionary are ignored. Unfortunately the Camembert model depends on overwriting `<unk>`, `<s>` and `</s>`.

The proposed solution here is to allow the dictionary to have entries like:
```
<unk> 999 #fairseq:overwrite
<s> 999 #fairseq:overwrite
</s> 999 #fairseq:overwrite
, 999
▁de 999
. 999
(...)
```

These will preserve the old overwriting behavior. Thus we can release a new `camembert.v0.tar.gz` with a dictionary like above and it works.
Pull Request resolved: fairinternal/fairseq-py#1073

Reviewed By: kahne

Differential Revision: D20284569

Pulled By: myleott

fbshipit-source-id: bf78fbff13c94bf8a6485cbdda62305ddc30c056
louismartin pushed a commit to louismartin/fairseq that referenced this issue Mar 24, 2020
@facebook-github-bot
Allow dictionaries to overwrite entries with #fairseq:overwrite comme…
901888c 
…nt (facebookresearch#1073)

Summary:
[This commit](facebookresearch@dd1298e) made it so that duplicate entries in a dictionary are ignored. Unfortunately the Camembert model depends on overwriting `<unk>`, `<s>` and `</s>`.

The proposed solution here is to allow the dictionary to have entries like:
```
<unk> 999 #fairseq:overwrite
<s> 999 #fairseq:overwrite
</s> 999 #fairseq:overwrite
, 999
▁de 999
. 999
(...)
```

These will preserve the old overwriting behavior. Thus we can release a new `camembert.v0.tar.gz` with a dictionary like above and it works.
Pull Request resolved: fairinternal/fairseq-py#1073

Reviewed By: kahne

Differential Revision: D20284569

Pulled By: myleott

fbshipit-source-id: bf78fbff13c94bf8a6485cbdda62305ddc30c056
moussaKam pushed a commit to moussaKam/language-adaptive-pretraining that referenced this issue Sep 29, 2020
@facebook-github-bot
Allow dictionaries to overwrite entries with #fairseq:overwrite comme…
7130145 
…nt (facebookresearch#1073)

Summary:
[This commit](facebookresearch@dd1298e) made it so that duplicate entries in a dictionary are ignored. Unfortunately the Camembert model depends on overwriting `<unk>`, `<s>` and `</s>`.

The proposed solution here is to allow the dictionary to have entries like:
```
<unk> 999 #fairseq:overwrite
<s> 999 #fairseq:overwrite
</s> 999 #fairseq:overwrite
, 999
▁de 999
. 999
(...)
```

These will preserve the old overwriting behavior. Thus we can release a new `camembert.v0.tar.gz` with a dictionary like above and it works.
Pull Request resolved: fairinternal/fairseq-py#1073

Reviewed By: kahne

Differential Revision: D20284569

Pulled By: myleott

fbshipit-source-id: bf78fbff13c94bf8a6485cbdda62305ddc30c056
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ngoyal2707 @yucoian @lematt1991