Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Inconsistency in GNoME's F1 Scores on Matbench #91

Closed
shrshr111 opened this issue Feb 20, 2024 · 1 comment · Fixed by #92
Closed

Inconsistency in GNoME's F1 Scores on Matbench #91

shrshr111 opened this issue Feb 20, 2024 · 1 comment · Fixed by #92
Labels
bug Something isn't working site Website related

Comments

@shrshr111
Copy link

I noticed that the F1 scores for GNoME listed on two different web pages within the Matbench Discovery section appear to be inconsistent:

https://matbench-discovery.materialsproject.org/models
and
https://matbench-discovery.materialsproject.org/

Could someone please clarify why there is a difference in the reported F1 scores for GNoME? Thanks!
janosh added a commit that referenced this issue Feb 20, 2024
@janosh
show model-stats-uniq-protos.json on /models to fix mismatch with lan…
766fb15 
…ding page metrics (closes #91)
@janosh
Copy link
Owner

janosh commented Feb 20, 2024

thanks for reporting! that is indeed a mistake. the model page was still showing metrics for the complete test set whereas the landing page recently transitioned to showing metrics on a subset of the WBM test set restricted to new and unique structures prototypes. see #75 for details.

@janosh janosh added bug Something isn't working site Website related labels Feb 20, 2024
@janosh janosh mentioned this issue Feb 20, 2024
Merged
janosh added a commit that referenced this issue Feb 20, 2024
@janosh
Fix landing page vs model page metrics mismatch (#92)
e596392 
* breaking: LabelEnum.dict() -> val_dict(), add label_dict() method

* add training_set col of main metrics table into urls

* fix gnome.yml targets: EFS-> EF

also add missing predictions note

* test mbd/enums.py and LabelEnum

* show model-stats-uniq-protos.json on /models to fix mismatch with landing page metrics (closes #91)

* render missing_preds notes in ModelCard tooltip

* rename model schema .yml + d.ts files

* clickable links to training sets in metrics table
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working site Website related
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix landing page vs model page metrics mismatch janosh/matbench-discovery
2 participants
@janosh @shrshr111