Skip to content

feat(multimodal): allow to template placeholders #3728

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 1 commit into from
Oct 4, 2024

Conversation

mudler
Copy link
Owner

@mudler mudler commented Oct 4, 2024

Description

This PR allows to customize the placeholders for the images that are sent back to the model.

For context, some models/libraries have different way to express images, videos or audio placeholders. For example, llama.cpp backend expects images within an [img-ID] tag, but other backends/models (e.g. vLLM) use a different notation ( <|image_|>). This changeset allow to specify the notation in the template config section of the YAML file:

For example, to override defaults, now it is possible to set in the model configuration the following:

template:
  video: "<|video_{{.ID}}|> {{.Text}}"
  image: "<|image_{{.ID}}|> {{.Text}}"
  audio: "<|audio_{{.ID}}|> {{.Text}}"

Notes for Reviewers

This PR is related to #3670 #3602 #2318

Signed commits

  • Yes, I signed my commits.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler added the enhancement New feature or request label Oct 4, 2024
Copy link

netlify bot commented Oct 4, 2024

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 03b3ce9
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/670006cb5994f80008c5e9ad
😎 Deploy Preview https://deploy-preview-3728--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudler mudler changed the title feat(multimodal): allow to template image placeholders feat(multimodal): allow to template placeholders Oct 4, 2024
@mudler mudler merged commit 648ffdf into master Oct 4, 2024
32 of 33 checks passed
@mudler mudler deleted the feat/multimodal_template branch October 4, 2024 16:32
siddimore pushed a commit to siddimore/LocalAI that referenced this pull request Oct 6, 2024
feat(multimodal): allow to template image placeholders

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant