fix: patch #1401

[cpacker]

https://platform.openai.com/docs/guides/text-generation/json-mode

JSON mode only supported by: gpt-4o, gpt-4-turbo, or gpt-3.5-turbo

Patch for #1401

[lenaxia]

            if "gpt-4o" in llm_config.model or "gpt-4-turbo" in llm_config.model or "gpt-3.5-turbo" in llm_config.model:
                data.response_format = {"type": "json_object"}

This defeats the point of my original change. If the model MUST be one of the openai ones, then response_format won't be set for other models, namely open source models.

I am using MemGPT with LocalAI (https://github.com/go-skynet/LocalAI/) and the models must receive the response_format={"type": "json_object"} in order to properly return JSON to memgpt. For instance, llama3, or Hermes 2 Pro all require this to work with memgpt.

Because MemGPT relies 100% on function calls it should be enabled by default because otherwise models won't return valid JSON. If you prefer it to be toggleable then I can put it behind an env var.

[cpacker]

            if "gpt-4o" in llm_config.model or "gpt-4-turbo" in llm_config.model or "gpt-3.5-turbo" in llm_config.model:
                data.response_format = {"type": "json_object"}

This defeats the point of my original change. If the model MUST be one of the openai ones, then response_format won't be set for other models, namely open source models.

I am using MemGPT with LocalAI (https://github.com/go-skynet/LocalAI/) and the models must receive the response_format={"type": "json_object"} in order to properly return JSON to memgpt. For instance, llama3, or Hermes 2 Pro all require this to work with memgpt.

Because MemGPT relies 100% on function calls it should be enabled by default because otherwise models won't return valid JSON. If you prefer it to be toggleable then I can put it behind an env var.

Ah OK sorry I misread the first PR - the MemGPT CI tests use GPT-4, so with the prior PR the tests were failing (since JSON mode isn't supported). Probably the easiest way to support both your workflow and have the default OAI GPT-4 setup pass is to use an env var like you mentioned? Happy to merge a PR for that in ASAP (or consider other ideas too).

[lenaxia]

Sounds good. I'll get a PR opened shortly. I'm not in a huge rush as I'm working on another PR to create an OpenAI compatible completions endpoint to make integration easier, so I've got the json_object set in my branch for now.