Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[Evaluation] LM Harness refactor #1410

Merged
merged 4 commits into from
Feb 8, 2025
Merged

[Evaluation] LM Harness refactor #1410

merged 4 commits into from
Feb 8, 2025

Conversation

kaisopos
Copy link
Contributor

@kaisopos kaisopos commented Feb 7, 2025

Description

Refactoring our LM Harness integration code (specifically the mapping and setting of lm_eval.simple_evaluate's model_args, in preparation for resolving GH issue 1358 (support for Remote vLLM with LM Harness)

Why the refactor?
The model args are added all over the place (in model_params.py, in lm_harness.py at evaluate and _create_extra_lm_harness_model_params_for_vlm) and it's becoming increasingly buggy to track what has been added for 5 different engines and 2 modalities. Thus, aggregated everything in a single function, documented what params the 4-5 LH Harness engines currently support, the mapping from Oumi's var names, etc.

Related issues

Fixes OPE-940

Before submitting

  • This PR only changes documentation. (You can ignore the following checks in that case)
  • Did you read the contributor guideline Pull Request guidelines?
  • Did you link the issue(s) related to this PR in the section above?
  • Did you add / update tests where needed?

Reviewers

At least one review from a member of oumi-ai/oumi-staff is required.

@kaisopos kaisopos requested review from taenin and xrdaukar February 7, 2025 22:07
Copy link
Collaborator

@taenin taenin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to update any tests for this change?

@kaisopos kaisopos merged commit 4ad8780 into main Feb 8, 2025
2 checks passed
@kaisopos kaisopos deleted the kostas/lm_harness_refactor branch February 8, 2025 04:38
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants