Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Add support for latest txtai indexing options #61

Closed
davidmezzetti opened this issue Dec 15, 2022 · 0 comments
Closed

Add support for latest txtai indexing options #61

davidmezzetti opened this issue Dec 15, 2022 · 0 comments
Assignees
@davidmezzetti
Milestone
v2.1.0

Comments

@davidmezzetti
Copy link
Member

Make the following improvements to better integrate with the latest txtai indexing options.

  • Only tokenize text if scoring is set. Transformers models don't need text tokenization and it can hurt performance.
  • Only filter sections when indexing if scoring is set. This isn't necessary for Transformers models.
  • Check the return type from embeddings.search and handle both dict content=true and tuple content=false return types.
@davidmezzetti davidmezzetti added this to the v2.1.0 milestone Dec 15, 2022
@davidmezzetti davidmezzetti self-assigned this Dec 15, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@davidmezzetti davidmezzetti

Labels
None yet
Projects
None yet
Milestone
v2.1.0
Development

No branches or pull requests
1 participant
@davidmezzetti