Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

fixing reproducibility of lmeval tests #1220

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fixing reproducibility of lmeval tests #1220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8b90a8e
attempts to fix lmeval failing tests
brian-dellabetta Mar 4, 2025
303ac43
pin random seed to resolve stochastic behavior
brian-dellabetta Mar 5, 2025
9ce3a0a
only set torch random seed
brian-dellabetta Mar 6, 2025
2d2e220
change setting of random seed to configuration
brian-dellabetta Mar 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,12 @@
"datasets",
"accelerate>=0.20.3,!=1.1.0",
"pynvml",
"compressed-tensors"
if version_info.build_type == "release"
else "compressed-tensors-nightly",
"pillow",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this a dev dependency rather than a base dependency?

(
"compressed-tensors"
if version_info.build_type == "release"
else "compressed-tensors-nightly"
),
],
extras_require={
"dev": [
Expand Down
2 changes: 1 addition & 1 deletion tests/lmeval/configs/vl_fp8_dynamic_per_token.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ cadence: weekly
model: Qwen/Qwen2-VL-2B-Instruct
model_class: TraceableQwen2VLForConditionalGeneration
scheme: FP8_DYNAMIC
seed: 42 #compressed model is sensitive to random seed
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit

Suggested change
seed: 42 #compressed model is sensitive to random seed
seed: 42 # compressed model is sensitive to random seed

lmeval:
model: "hf-multimodal"
model_args:
Expand All @@ -10,7 +11,6 @@ lmeval:
convert_img_format: True
task: mmmu_val_economics
num_fewshot: 0
limit: 1000
batch_size: 8
metrics:
acc,none: 0.333
2 changes: 1 addition & 1 deletion tests/lmeval/configs/vl_int8_w8a8_dynamic_per_token.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ scheme: INT8_dyn_per_token
recipe: tests/e2e/vLLM/recipes/INT8/recipe_int8_channel_weight_dynamic_per_token.yaml
dataset_id: lmms-lab/flickr30k
dataset_split: "test[:512]"
seed: 42 #compressed model is sensitive to random seed
lmeval:
model: "hf-multimodal"
model_args:
Expand All @@ -13,7 +14,6 @@ lmeval:
convert_img_format: True
task: mmmu_val_economics
num_fewshot: 0
limit: 1000
metrics:
acc,none: 0.233
batch_size: 8
4 changes: 2 additions & 2 deletions tests/lmeval/configs/vl_w4a16_actorder_weight.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ recipe: tests/e2e/vLLM/recipes/actorder/recipe_w4a16_actorder_weight.yaml
dataset_id: lmms-lab/flickr30k
dataset_split: "test[:512]"
scheme: W4A16_actorder_group
seed: 42 #compressed model is sensitive to random seed
lmeval:
model: "hf-multimodal"
model_args:
Expand All @@ -13,7 +14,6 @@ lmeval:
convert_img_format: True
task: mmmu_val_economics
num_fewshot: 0
limit: 1000
metrics:
acc,none: 0.4
acc,none: 0.366
batch_size: 4
8 changes: 8 additions & 0 deletions tests/lmeval/test_lmeval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
import os
import random
import shutil
from pathlib import Path

import numpy
import pytest
import torch
import yaml
from loguru import logger
from pydantic import BaseModel
Expand Down Expand Up @@ -73,6 +75,12 @@ def set_up(self):
self.quant_type = eval_config.get("quant_type")
self.save_dir = eval_config.get("save_dir")

seed = eval_config.get("seed", None)
if seed is not None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use from transformers import set_seed
https://github.com/huggingface/transformers/blob/94ae1ba5b55e79ba766582de8a199d8ccf24a021/src/transformers/trainer_utils.py#L92

random.seed(seed)
numpy.random.seed(seed)
torch.manual_seed(seed)

logger.info("========== RUNNING ==============")
logger.info(self.scheme)

Expand Down
Loading