Getting frequent replicate.exceptions.ModelError on small, text only queries using python API, llama-3-70b-instruct model

[gthaker]

I get this error. If I rerun it usually goes thru fine but dies again down the road. (I am trying 300 total question from a professional exam.) I had no trouble running this with llama-2-70b model, i did it many times.

Traceback (most recent call last):
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/run_llm.py", line 58, in <module>
    key, answer, delta = ask_llama_2(ques)
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/llama_2_api.py", line 35, in ask_llama_2
    output = replicate.run(model,
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/venv/lib/python3.10/site-packages/replicate/client.py", line 157, in run
    return run(self, ref, input, **params)
  File "/export/home/gthaker/repos/radonc_inSvc_LLM/venv/lib/python3.10/site-packages/replicate/run.py", line 61, in run
    raise ModelError(prediction.error)
replicate.exceptions.ModelError

[mattt]

Hi @gthaker. I'm sorry that you're occasionally having issues running meta/meta-llama-3-70b-instruct. Can you share a link to or logs for any failed predictions?

replicate.exceptions.ModelError indicates a problem with the model, so there's nothing to be done with the client itself. What you could try to do is wrap your call in a try/catch and implement retry logic to suit your needs.

[gthaker]

Can I send you the link in a private message? Prob. best not share the query with everyone. GHT

…

On Sun, Jun 2, 2024 at 5:49 AM Mattt ***@***.***> wrote: Hi @gthaker <https://github.com/gthaker>. I'm sorry that you're occasionally having issues running meta/meta-llama-3-70b-instruct. Can you share a link to or logs for any failed predictions? replicate.exceptions.ModelError indicates a problem with the model, so there's nothing to be done with the client itself. What you could try to do is wrap your call in a try/catch and implement retry logic to suit your needs. — Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC7QKFDOBCNV7UHGBW5IAJLZFLTEHAVCNFSM6AAAAABIUKYSPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTG43TOOJWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Y-Sui]

Have the same issue here. The error still exists not only for meta/meta-llama-3-70b-instruct, but also for model mixtral-8x7b-instruct-v0.1.