Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fix a bug in run_qa_no_trainer.py #16867

Conversation

dreamgonfly
Copy link
Contributor

What does this PR do?

Fixes a bug in run_qa_no_trainer.py.
In the code, eval_metric dictionary generated from squad metric doesn't have 'exact' key.
KeyError: 'exact' raised during execution.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed.

eval_metric doesn't have 'exact' key.
KeyError: 'exact' raised during execution.
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@dreamgonfly dreamgonfly changed the title Update run_qa_no_trainer.py Fix a bug in run_qa_no_trainer.py Apr 22, 2022
@searchivarius
Copy link
Contributor

searchivarius commented Apr 27, 2022

Hi @dreamgonfly I have a more comprehensive PR that fixes this issue. Turns out that saving of the metrics is tricky, because the unit tests (which fail in your case) actually use SQuAD v2 metric and they have a different name for the exact matching score. Naturally, you were running this code from the command line and it was using the SQuAD v1 metric. This is why your tests are failing, but the command line results are fine (it took me a few hours of head scratching to figure this out):
#16958

@dreamgonfly dreamgonfly closed this May 4, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants