Run locally?

[CHesketh76]

Was just wondering if i can run this locally?

[sammcj]

Was trying to figure this out too, surely it must be possible without a togetherai api key?

[sammcj]

Looks like it's currently hardcoded to use togetherai, but actually only needs any openai compatible endpoint: https://github.com/togethercomputer/MoA/blob/main/utils.py#L28

[CHesketh76]

@sammcj that's what I was thinking as well

[sammcj]

Something like this I think #12

[bolecki]

i had a quick go at this as well over here: #13

it is not very configurable, but if you have an openai compatible BE hosting several models, you can just set OPENAI_API_BASE and update the model names in bot.py. i was using LM Studio and added additional models in playground. depending on your hardware, might be tough to get reasonable models to fit in VRAM.

should be easy to split out a streaming call for the aggregator as well.

[sammcj]

Got it all working - https://github.com/sammcj/MoA

[Boubou78000]

#17

[yananchen1989]

@IsThatYou hi, for the function https://github.com/togethercomputer/MoA/blob/main/utils.py#L99, in fact, it is a compatiable openai api for backend fastchat or vllm, which supports various open-llms, am I right ? It is not a true original OpenAI ?

[IsThatYou]

@IsThatYou hi, for the function https://github.com/togethercomputer/MoA/blob/main/utils.py#L99, in fact, it is a compatiable openai api for backend fastchat or vllm, which supports various open-llms, am I right ? It is not a true original OpenAI ?

That's correct. We haven't tested with other APIs but it should be good with most openai compatible api.