OpenAI Vision and TTS

[mudler]

https://openai.com/blog/new-models-and-developer-products-announced-at-devday

OpenAI recently announced Turbo vision and text to speech API. This card is to create an API equivalent for the new functionalities

• New GPT-4 Turbo Vision API -> LLaVA https://platform.openai.com/docs/guides/vision
• JSON output in the object type
• call multiple functions in one message #1275
• New text-to-speech API  #1276

For Turbo vision we need first #1054

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

LocalAI already supports text generation with GPTs using llama.cpp and other backends (such as rwkv.cpp). For the new GPT-4 Turbo Vision API, we can add a new endpoint to LocalAI that uses the LLaVA library. We will need to install the LLaVA library and set up the backend in the codebase.
For the text-to-speech API, we can use the OpenAI functions provided by LocalAI to generate audio from text. We can add a new endpoint to LocalAI that takes text input and returns an audio file of the generated speech. We will need to set up the backend in the codebase and train the model on a dataset of text-to-speech pairs.
To get started, we can create a feature request for each of these features on the LocalAI GitHub issue tracker and assign them to the relevant team members.

Sources:

• https://localai.io/features/index.html
• https://localai.io/features/audio-to-text/index.html
• https://localai.io/index.html
• https://localai.io/features/text-generation/index.html

[mudler]

GPT Vision support API is introduced as part of #1254, along with JSON mode (https://platform.openai.com/docs/guides/text-generation/json-mode)

[mudler]

On TTS: By luck I think we are already compatible, but I've didn't tried myself the client

[mudler]

Closing as the big portion of this is done, created follow-ups for what's left