Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Update docs to add notes about BEIR uniCOIL and HC4 regressions #1915

Merged
merged 1 commit into from
Jun 19, 2022

Conversation

lintool
Copy link
Member

@lintool lintool commented Jun 18, 2022

% python src/main/python/beir/gather_beir_results.py 
nDCG@10                  flat    multi   UCx     SPLADE
                         ------  ------  ------  ------
trec-covid               0.5947  0.6559  0.6403  0.7109
bioasq                   0.5225  0.4646  0.4771  0.5035
nfcorpus                 0.3218  0.3254  0.3325  0.3454
nq                       0.3055  0.3285  0.4249  0.5442
hotpotqa                 0.6330  0.6027  0.6671  0.6860
fiqa                     0.2361  0.2361  0.2888  0.3514
signal1m                 0.3304  0.3304  0.2750  0.2957
trec-news                0.3952  0.3977  0.3743  0.3936
robust04                 0.4070  0.4070  0.4030  0.4581
arguana                  0.3970  0.4142  0.3959  0.5210
webis-touche2020         0.4422  0.3673  0.2976  0.2435
cqadupstack-android      0.3801  0.3709  0.3563  0.3954
cqadupstack-english      0.3453  0.3321  0.3635  0.4026
cqadupstack-gaming       0.4822  0.4418  0.4459  0.5061
cqadupstack-gis          0.2901  0.2904  0.2713  0.3223
cqadupstack-mathematica  0.2015  0.2046  0.2106  0.2423
cqadupstack-physics      0.3214  0.3248  0.3121  0.3668
cqadupstack-programmers  0.2802  0.2963  0.2967  0.3412
cqadupstack-stats        0.2711  0.2790  0.2860  0.3142
cqadupstack-tex          0.2244  0.2086  0.2279  0.2575
cqadupstack-unix         0.2749  0.2788  0.2826  0.3292
cqadupstack-webmasters   0.3059  0.3008  0.2974  0.3343
cqadupstack-wordpress    0.2483  0.2562  0.2573  0.2839
quora                    0.7886  0.7886  0.6625  0.8136
dbpedia-entity           0.3180  0.3128  0.3383  0.4416
scidocs                  0.1490  0.1581  0.1439  0.1590
fever                    0.6513  0.7530  0.8118  0.7962
climate-fever            0.1651  0.2129  0.1824  0.2276
scifact                  0.6789  0.6647  0.6856  0.6992
                         ------  ------  ------  ------
                         0.4244  0.4288  0.4279  0.4740

@codecov-commenter
Copy link

Codecov Report

Merging #1915 (c0717df) into master (f592832) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1915   +/-   ##
=========================================
  Coverage     58.67%   58.67%           
  Complexity     1079     1079           
=========================================
  Files           180      180           
  Lines         10420    10420           
  Branches       1429     1429           
=========================================
  Hits           6114     6114           
  Misses         3786     3786           
  Partials        520      520           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f592832...c0717df. Read the comment docs.

@lintool lintool merged commit 308138e into master Jun 19, 2022
@lintool lintool deleted the regression-updates branch June 19, 2022 00:45
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants