Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Port SparseML Remote Code Fix #21

Merged
merged 1 commit into from
Jul 12, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion src/llmcompressor/transformers/sparsification/sparse_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,17 +74,29 @@ def skip(*args, **kwargs):
)

# instantiate compressor from model config
compressor = ModelCompressor.from_pretrained(pretrained_model_name_or_path)
compressor = ModelCompressor.from_pretrained(
pretrained_model_name_or_path, **kwargs
)

# temporarily set the log level to error, to ignore printing out long missing
# and unexpected key error messages (these are EXPECTED for quantized models)
transformers_logger = logging.getLogger("transformers.modeling_utils")
restore_log_level = transformers_logger.getEffectiveLevel()
transformers_logger.setLevel(level=logging.ERROR)

if kwargs.get("trust_remote_code"):
# By artifically aliasing
# class name SparseAutoModelForCausallLM to
# AutoModelForCausalLM we can "trick" the
# `from_pretrained` method into properly
# resolving the logic when
# (has_remote_code and trust_remote_code) == True
cls.__name__ = AutoModelForCausalLM.__name__

model = super(AutoModelForCausalLM, cls).from_pretrained(
pretrained_model_name_or_path, *model_args, **kwargs
)

if model.dtype != model.config.torch_dtype:
logger.warning(
f"The dtype of the loaded model: {model.dtype} is different "
Expand Down
Loading