RAPTOR failing [ERROR]Fail to bind LLM used by RAPTOR: 3 vs. 4

[sandstad]

I'm struggling with the stability of RAPTOR. I have just enabled RAPTOR on the Knowledge Base and not tuned any parameters or prompt.

I'm running Snowfflake as the Embedding LLM and Quen2.5:7b as the Chat LLM:

$ ollama ps
NAME                              ID              SIZE      PROCESSOR    UNTIL
qwen2.5:7b-ctx8k                  6fb8503ba6cd    10 GB     100% GPU     Forever
snowflake-arctic-embed2:latest    5de93a84837d    1.7 GB    100% GPU     4 minutes from now
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.40.07              Driver Version: 551.52         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070        On  |   00000000:01:00.0  On |                  N/A |
| 33%   60C    P2            148W /  200W |   11808MiB /  12282MiB |     89%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        33      G   /Xwayland                                   N/A      |
|    0   N/A  N/A      1344      C   /python3.10                                 N/A      |
|    0   N/A  N/A     26709      C   /ollama_llama_server                        N/A      |
|    0   N/A  N/A     75688      C   /ollama_llama_server                        N/A      |
+-----------------------------------------------------------------------------------------+

RAPTOR FAIL.txt

Attached the task_executor logs and a screenshot of the error in the GUI. Seems that the LLM is respondig to all requests with a "HTTP/1.1 200 OK".

Some times the I get a success when I just restart the process. Some times it fails again. Any suggestions on how to get this stable?

[sandstad]

Here is an example where I just have restarted the job without deleteing the chunks. Same file as before

RAPTOR SUCCESS.txt