Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Graceful exit or validation of server parameters #89

Open
thameem-abbas opened this issue Feb 1, 2025 · 0 comments
Open

Graceful exit or validation of server parameters #89

thameem-abbas opened this issue Feb 1, 2025 · 0 comments

Comments

@thameem-abbas
Copy link
Collaborator

thameem-abbas commented Feb 1, 2025 •
edited
Loading

We should include a check of server parameters before launching the requests and letting all of them fail. For example, when the server does not serve the model name specified, we run into errors like attached below. Some of the basic checks:

  • Is the server alive ?
  • Does the server have the endpoint we're looking for ?
  • Is the server serving the model we are requesting ?

Longshot goal: Identify successive failures of same reason and terminate all execution. To identify if the server dies or becomes unreachable mid-test

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-92 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-25 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-61 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-15 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-97 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-109 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-88 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-46 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-62 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-90 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-57 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-53 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-103 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-69 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-42 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-94 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-16 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-14 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,846 ERROR    user SpawnProcess-85 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,845 ERROR    user SpawnProcess-32 Connection error
Traceback (most recent call last):
  File "/src/llm-load-test/plugins/openai_plugin.py", line 225, in streaming_request_http
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.128.1.163:8000/v1/completions
2025-02-01 14:13:12,889 INFO     root MainProcess Adding 128 entries to dataset queue
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@thameem-abbas