Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[QUESTION] MQM dataset on huggingface #227

Open
aenaliph opened this issue Aug 16, 2024 · 0 comments
Open

[QUESTION] MQM dataset on huggingface #227

aenaliph opened this issue Aug 16, 2024 · 0 comments
Labels
question Further information is requested

Comments

@aenaliph
Copy link

For the dataset shared here: https://huggingface.co/datasets/RicardoRei/wmt-mqm-human-evaluation

the data summary says the score is: MQM score. Sample row below:

1776 en-de He said: "I know of several other guys over the internet who feel the same way," but added that they are "too cowardly to act on their anger." Er sagte: „Ich weiß ganz genau, dass es noch mehr Typen im Internet gibt, die das Gleiche denken wie ich“, so Minassian, wenngleich er hinzufügte, dass diese „wohl zu feige wären, um ihrem Zorn freien Lauf zu lassen“. Er sagte: „Ich kenne mehrere andere Typen über das Internet, die genauso empfinden“, fügte aber hinzu, dass sie „zu feige sind, um ihre Wut auszuleben“. -0.333333 Human-A.0 3 news 2020

  • Is the score here in bold a z-score already normalized per annotator?
  • If so, does it make sense to combine this MQM dataset with the DA dataset to train a COMET-like model from scratch?
@aenaliph aenaliph added the question Further information is requested label Aug 16, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant