Support for locally hosted LLMs

[Schallabajzr]

I was following a blog post and self hosting both:

• alpaca.13b.ggmlv3
• ggml-gpt4all-j

the requests to these servers work via curl as per this
curl http://localhost:8080/v1/models {"object":"list","data":[{"id":"gpt-3.5-turbo","object":"model"}]}%

but when I try to communicate with it through the text generator plugin with the model rename to gpt-3.5-turbo to the self hosted instance (running inside docker at localhost:8080) I get the following error.
`

[!failure]- Failure
TypeError: Failed to fetch

• plugin:obsidian-textgenerator-plugin:57307 eval
  plugin:obsidian-textgenerator-plugin:57307:9

• new Promise

• plugin:obsidian-textgenerator-plugin:57291 TextGenerator.eval
  plugin:obsidian-textgenerator-plugin:57291:14

• Generator.next

• plugin:obsidian-textgenerator-plugin:78 eval
  plugin:obsidian-textgenerator-plugin:78:61

• new Promise

• plugin:obsidian-textgenerator-plugin:62 __async
  plugin:obsidian-textgenerator-plugin:62:10

• plugin:obsidian-textgenerator-plugin:57290 TextGenerator.request
  plugin:obsidian-textgenerator-plugin:57290:12

• plugin:obsidian-textgenerator-plugin:57275 TextGenerator.eval
  plugin:obsidian-textgenerator-plugin:57275:86

• Generator.next

`

[haouarihk]

The support for locally hosted llms are coming soon.
it is in the beta release,
it will be out soon

[dahifi]

+1 on this as well. LMStudio support with this will be awesome.

Any related tasks on this @haouarihk, or are you doing all this in private?

[haouarihk]

+1 on this as well. LMStudio support with this will be awesome.

Any related tasks on this @haouarihk, or are you doing all this in private?

i forgot to mention it there, LMStudio support is out,
if you want up to date tasks. you can join our discord community

[dahifi]

Relevant Discord link here, below is copypasta from the thread, c/o haitam(526577079509057537). https://discord.com/channels/1083485983879741572/1159894948636799126

• step 1: Download LM Studio, and install a model of your choice
• step 2: Start the server, with CORS enabled, and don't forget to copy the endpoint as shown in the image
  
  step 3: go to the plugin settings and select Openai either chat or instruct, and paste in the new base path
  

note that it will not work without api key being set to something,
we recommend setting it to random text like "asdf"
if the generation is slow, and you have a decent gpu
make sure to set these