NoneType Error in result.output_text Join When Serving Qwen2.5-72B-Instruct

[h-albert-lee]

NoneType Error in result.output_text Join When Serving Qwen2.5-72B-Instruct with vLLM on H100 8 GPUs

Description

When serving Qwen2.5-72B-Instruct with vLLM on an H100 8-GPU setup, I encountered an error during load testing with the sample dataset file openorca_large_subset_011.jsonl (from the existing datasets). The error occurs at the following line:

result.output_text = "".join(token['text'] or "" for token in tokens)

with the message:

expected str instance, NoneType found

This indicates that one or more tokens have None as their 'text' value.

Steps to Reproduce

1. Serve Qwen2.5-72B-Instruct using vLLM on an H100 8-GPU system.
2. Run a load test using the sample dataset file openorca_large_subset_011.jsonl.
3. Observe that during processing, the code fails at the join operation due to a None value in one or more tokens.

Expected Behavior

All tokens should contain a valid string in their 'text' field, or the code should handle None values gracefully. Using token['text'] or "" as a workaround avoids the error, but it’s unclear if this approach aligns with the original de#tentions.

Actual Behavior

The join operation fails with the error:

expected str instance, NoneType found

indicating that at least one token’s 'text' value is None.

Workaround

Changing the code to:

result.output_text = "".join(token['text'] or "" for token in tokens)

prevents the error. However, I’m not sure if this is the intended solution or if the underlying issue of receiving None for some tokens should be addressed differently.

Additional Context

• Model: Qwen2.5-72B-Instruct
• Hardware: H100 (8 GPUs)
• Dataset: openorca_large_subset_011.jsonl

Request

Could you please confirm whether handling None values in this manner is acceptable, or if there’s a different intended approach for processing tokens? Any insights into whether the plugin should filter out or replace None values would be appreciated.

Thank you!