json_func_calls_with_inner_thoughts string broken?

[jimlloyd]

Describe the bug
Perhaps I am mistaken, but it seems to me that the string production rule in json_func_calls_with_inner_thoughts.gbnf is not able to represent strings that contain quote " characters, square bracket characters [] or braces {}.

The string production rule in json.gbnf is complex because it encodes the true requirements for strings in JSON. The simpler string production rule in json_func_calls_with_inner_thoughts.gbnf excludes large classes of legal strings. I think this means that functions such as conversation_search cannot be legal grammar when conversation includes code snippets with braces (e.g. C++ code).

Likewise, I think the simplified ws production rule is also a problem. The ws in json.gbnf is a rule that matches white space, but the production rule in json_func_calls_with_inner_thoughts.gbnf actually only matches the empty string.

Please correct me if I am wrong.

Please describe your setup

N/A

Screenshots
N/A

Additional context
N/A

---

If you're not using OpenAI, please provide additional information on your local LLM setup:

Local LLM details

If you are trying to run MemGPT with local LLMs, please provide the following information:

• The exact model you're trying to use (e.g. dolphin-2.1-mistral-7b.Q6_K.gguf)
• The local LLM backend you are using (web UI? LM Studio?)
• Your hardware for the local LLM backend (local computer? operating system? remote RunPod?)

[jimlloyd]

I have been trying to gather empirical evidence that this bug causes errors with function calling. I modified the json_func_calls_with_inner_thoughts.gbnf file to use the string and ws rules from json.gbnf and the result in the first test seemed fantastic. This is the dialog:

That was by far the cleanest interaction I've had recently when trying to have conversations about C++ code.

I next did a git stash so that the change was reverted, and tried the same leading instruction again and got poor results. So I did a git stash pop and ran the test again. And was unable to reproduce the clean result. (I used a new agent each time).

I made multiple trials and never was able to reproduce the good result.

I think I need to trace through the code and figure out how to get some useful debug logging.

[jimlloyd]

FYI this was with NexusRaven-V2-13B.

[netixc]

can u give the adjusted .gbnf file i would like to test it

[jimlloyd]

@netixc see #754.