Support third-party AI providers

[OiAnthony]

Description

Support third-party AI providers

• Support customizing those that conform to the OpenAI API specification, such as custom endpoint and APIKey.
• Provide an adapter to facilitate the access of third-party AI that does not conform to the OpenAI API specification.

Use case

Users can choose local or third-party AI providers

Anything else?

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[MingStudentSE]

I think that is a good idea.

[EYHN]

Thank you for your suggestion on supporting third-party AI providers.

We are currently focused on optimizing the user experience and will not be adding this feature in the short term. However, we plan to support custom AI providers in the future.

[EYHN]

Since this won't be achieved in the short term, I will convert this issue into a discussion so that we can continue to discuss it.

[MFYDev]

Good idea, will need this feature as well

[ProfessorDey]

There's no reason this shouldn't already be a thing when the OpenAI NodeJS module has had access to a 'baseURL' parameter for ages and even provide explicit examples. With them having reimplemented half the classes with new random names (Seriously why even bother with 'copilot' when it just adds confusion with Microsoft's Copilot?). its taking a bit to find exactly where the config is being pulled from but the same config value should still be directly passable to the module.
https://github.com/openai/openai-node/tree/v4#configuring-an-https-agent-eg-for-proxies
https://github.com/openai/openai-node/blob/v4/examples/azure.ts

EDIT: Finally tracked things down and yes, they're literally just pulling directly from the /root/.affine/self-host/config/affine.js file it seems. The correct addition according to text-generation-webui is:

AFFiNE.use('copilot', {
  openai: {
    baseURL: 'http://127.0.0.1:5000/',
    apiKey: 'sk-111111111111111111111111111111111111111111111111',
  }

But I get a connection error, so I'm not quite sure what other info is missing. It is at least clearly redirecting things, it's likely just either a TLS issue or more likely an issue with needing a bit more added to the config.

[ProfessorDey]

Following up on this, using TCP dump, it oddly appears as though its actually trying to send code directly (Bearing in mind I'm running on canary, so this could just be an internal code issue) to graphQL, so I'm not really sure how this is operating in general:

02:42:16.500637 IP 172.18.0.1.56600 > 172.18.0.4.gw: Flags [P.], seq 3621119956:3621121809, ack 4262905396, win 501, options [nop,nop,TS val 2675219065 ecr 275298778], length 1853
E..q..@.@.&r...................4...._......
.t.y.h..POST /graphql HTTP/1.1
Host: 0.0.0.0:3010
Connection: keep-alive
Content-Length: 495
x-operation-name: createCopilotMessage
DNT: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
x-definition-name: createCopilotMessage
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryecNZE0gnMI7ZsR1n
Accept: */*
Origin: http://0.0.0.0:3010
Referer: http://0.0.0.0:3010/workspace/cc0bb58c-dac9-4d33-b01f-4efd7b6e1dfa/yyH5j5PY96QgDhozna1YB
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: remember_token=cc0b0c1289f34351ae4afe92f3181b11|b8a4deac5173c78ba93894bf6078f9f6957431cef0bfcbb8f0a22ba8f33c1ad4590f624026079b59cde3dce722db9dd0ca8cee550bd1a91e707062bf0dd7a22c; session=.eJxdkUtuHDEMRO-iZTAdiKSoz1zFCBoURcWNeD5Qd68C392yvUp2qgKrUHj661bdR1-Pxx-7u6tjKlF7SAWUUQw5iXX0sSasZjnlxKXW2sV8JgrKPcbGLROjNrLqLm7tw_ZXdz3GaVNtbdYK9uw9QKTgSQsLENcSWjEu0IRKCz5NmxJLFi7Rd9aAJYUg0BoVQmytCytSs25NfVVvIbfckxc1SSl5skjoS-0ws5Yg9lhjxOz5c9a52_heozPtFTCXToEYxIJ0K9gJMlSAeV2HSdNx3ururi_u9XGz6SaYJDDwEiOXJWAOS2UPSxHJNjdaqp8Eftvdjk2XY8h2_K8XedvsvnST45yg3K-L--cHLNScWkpRA3udkChKaAxZEYwsQ4KKDHH26jmG3Y9118fTZvTH9Pq-qs73bse3eoq46wxhQMpAP3OOkAJe3E2eE8fMb32zMW8u7qto7dvb8WW8fwAgwJ4N.ZhJeJw.ZvCk3F-IxwcCS6U5-pd3EKYqRm8; affine_session=4dd94f58-ea31-46da-b0db-e312880bffc5

------WebKitFormBoundaryecNZE0gnMI7ZsR1n
Content-Disposition: form-data; name="operations"

{"query":"\nmutation createCopilotMessage($options: CreateChatMessageInput!) {\n  createCopilotMessage(options: $options)\n}","variables":{"options":{"sessionId":"57821d4c-cd0b-40b2-bd4f-2faf5340d676","content":"Testing"}},"map":{},"name":"createCopilotMessage"}
------WebKitFormBoundaryecNZE0gnMI7ZsR1n
Content-Disposition: form-data; name="map"

{}
------WebKitFormBoundaryecNZE0gnMI7ZsR1n--

[ProfessorDey]

So using edgeshark + wireshark live capture, I'm at least able to see the internal requests, but not the external ones, it's definitely working through the internal GraphQL API, but the way its doing it definitely seems odd. The path for a simple message being the following:
http://0.0.0.0:3010/api/copilot/chat/57821d4c-cd0b-40b2-bd4f-2faf5340d676/stream?messageId=8b3913b9-d7d1-4b6f-bf19-0b1e2efc545c

[ProfessorDey]

Success! Well, technically. turns out the remaining issue was simple. Since AFFiNE runs inside docker, just using localhost won't produce the right URL, you need to enter the actual internal network IP address of your PC in order for the request to route outside of the docker container and access the host system.

AFFiNE.use('copilot', {
  openai: {
    baseURL: 'http://192.168.0.98:5000/v1',
    apiKey: 'sk-111111111111111111111111111111111111111111111111',
  }

However it seems AFFiNE uses a rather long context, longer than my model settings supported:

14:18:41-949202 INFO     Loaded "llama2-13b-tiefighter.Q5_0.gguf" in 15.31 seconds.
14:18:41-951586 INFO     LOADER: "llama.cpp"
14:18:41-952090 INFO     TRUNCATION LENGTH: 4096
14:18:41-952610 INFO     INSTRUCTION TEMPLATE: "Alpaca"
14:19:17-722357 ERROR    Failed to build the chat prompt. The input is too long for the available context length.

                         Truncation length: 4096
                         max_new_tokens: 4096 (is it too high?)
                         Available context length: 0

Exception in ASGI application

The Full JSON of the query:

{
  "stream": true,
  "messages": [
    {
      "role": "system",
      "content": "You are AFFiNE AI, a professional and humorous copilot within AFFiNE. You are powered by latest GPT model from OpenAI and AFFiNE. AFFiNE is an open source general purposed productivity tool that contains unified building blocks that users can use on any interfaces, including block-based docs editor, infinite canvas based edgeless graphic mode, or multi-dimensional table with multiple transformable views. Your mission is always to try your very best to assist users to use AFFiNE to write docs, draw diagrams or plan things with these abilities. You always think step-by-step and describe your plan for what to build, using well-structured and clear markdown, written out in great detail. Unless otherwise specified, where list, JSON, or code blocks are required for giving the output. Minimize any other prose so that your responses can be directly used and inserted into the docs. You are able to access to API of AFFiNE to finish your job. You always respect the users' privacy and would not leak their info to anyone else. AFFiNE is made by Toeverything .Pte .Ltd, a company registered in Singapore with a diverse and international team. The company also open sourced blocksuite and octobase for building tools similar to Affine. The name AFFiNE comes from the idea of AFFiNE transform, as blocks in affine can all transform in page, edgeless or database mode. AFFiNE team is now having 25 members, an open source company driven by engineers."
    },
    {
      "role": "user",
      "content": "testing'"
    }
  ],
  "model": "gpt-4o",
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "temperature": 0.5,
  "max_tokens": 4096,
  "response_format": {
    "type": "text"
  },
  "user": "10a22515-2c1a-4a54-ac42-7aaf123a91ee"
}

[ProfessorDey]

It works! Haha! By using a model with at least an 8k context window, you can properly access the model, but of particular note is that for Text-Generation-Webui, you also need to create a new "None" character as there's an error handling requests that don't have a character value set. AFFiNE is directly setting the context rather than having a character applied, so adding a "None" character avoids the following error here:

14:42:36-340472 ERROR    Could not find the character "None" inside characters/. No character has been loaded.
Exception in ASGI application

For convenience, here's a mostly blank character you can just directly load up:
None.json

With the character it now works perfectly, which is awesome!

14:47:48-345619 INFO     New character saved to "characters/None.yaml".
14:49:38-261417 INFO     STOPPING_STRINGS=
['<|eot_id|>']

14:49:38-273597 INFO     PROMPT=
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are AFFiNE AI, a professional and humorous copilot within AFFiNE. You are powered by latest GPT model from OpenAI and AFFiNE. AFFiNE is an open source general purposed productivity tool that contains unified building blocks that users can use on any interfaces, including block-based docs editor, infinite canvas based edgeless graphic mode, or multi-dimensional table with multiple transformable views. Your mission is always to try your very best to assist users to use AFFiNE to write docs, draw diagrams or plan things with these abilities. You always think step-by-step and describe your plan for what to build, using well-structured and clear markdown, written out in great detail. Unless otherwise specified, where list, JSON, or code blocks are required for giving the output. Minimize any other prose so that your responses can be directly used and inserted into the docs. You are able to access to API of AFFiNE to finish your job. You always respect the users' privacy and would not leak their info to anyone else. AFFiNE is made by Toeverything .Pte .Ltd, a company registered in Singapore with a diverse and international team. The company also open sourced blocksuite and octobase for building tools similar to Affine. The name AFFiNE comes from the idea of AFFiNE transform, as blocks in affine can all transform in page, edgeless or database mode. AFFiNE team is now having 25 members, an open source company driven by engineers.<|eot_id|><|start_header_id|>user<|end_header_id|>

Can you read this?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

[ProfessorDey]

Best I can tell the content of the character context doesn't actually matter as AFFiNE directly overrides it anyway, will have to test the other use cases of the AI but the chat at least works.

[OiAnthony]

link: #7705

[HYQ199506]

If using Llama3 self built services, images, flowcharts, and other content cannot be used. How should I configure it.

[HYQ199506]

Yes, it would be great if you could use some multimodal models deployed by Xinference, such as QWEN-2.5-VL or multiple different models, combine them for configuration, and add a test button on the page to monitor if the configuration is available

[mdlmarkham]

It seems like this could be easily added to /admin...

[dadcoachengineer]

Pretty simple config to use my self-hosted Open WebUI instance.

How are you doing model selection for different AI functions? Is that stored in Postgres?