add list_evaluation_modules function
#59
Conversation
lvwerra
changed the title
list_evaluation_modules function
|
The documentation is not available anymore as the PR was closed or merged. |
src/evaluate/inspect.py
Outdated
| with_details (:obj:`bool`, optional, default ``False``): Return the full details on the metrics instead of only the short name. | ||
| type (:obj:`string`, optional, default ``None``): Type of evaluation modules to list. If ``None`` all types are listed. | ||
| include_community (:obj:`bool`, optional, default ``True``): Include community modules in the list. | ||
| with_details (:obj:`bool`, optional, default ``False``): Return the full details on the metrics instead of only the ID. |
There was a problem hiding this comment.
Maybe for consistency with huggingface_hub this should be renamed to full. No strong thoughts though.
src/evaluate/inspect.py
Outdated
| @@ -31,41 +32,67 @@ class SplitsNotFoundError(ValueError): | |||
| pass | |||
|
|
|||
|
|
|||
| def list_metrics(with_community_metrics=True, with_details=False): | |||
| def list_evaluation_modules(type=None, include_community=True, with_details=False): | |||
| """List all the metrics script available on the Hugging Face Hub. | |||
There was a problem hiding this comment.
Maybe you should clarify that this return a list of repo ids. The name tells me it returns a list of modules, which is a bit confusing since it really returns ids. Similar with the docstring
There was a problem hiding this comment.
I actually changed this a bit: for canonical modules the the namespace is omitted so this function returns what the user should put into load() function. Technically, load("evaluate-metric/accuracy") also works but I think we should use load("accuracy") as the standard.
|
|
||
| r = requests.get(HF_LIST_ENDPOINT.format(type=type)) | ||
| r.raise_for_status() | ||
| d = r.json() |
There was a problem hiding this comment.
At one point it would be cool to have list_spaces in huggingface_hub. I opened huggingface/huggingface_hub#881
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
Updates the
list_metricsfunction tolist_evaluation_modules. It uses thehttps://huggingface.co/api/spaces?filter={type}endpoint to query all available modules from the hub.cc @julien-c