Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add arguments to pass document parsing and chunking options to google_discovery_engine_data_store #17554

Open
etekin-amfam opened this issue Mar 12, 2024 · 2 comments

Comments

@etekin-amfam
Copy link

etekin-amfam commented Mar 12, 2024

Google's VertexAI Search & Conversation Data Stores have further options to refine how documents are ingested that can significantly improve performance. However, these options need to be passed at creation, and are currently not supported by terraform. As such, we're having to revert to deploying these data stores manually instead of terraform., making things harder to manage. Adding these options to the terraform module so we can deploy data stores via terraform would make maintenance significantly easier and less manual.
For the arguments I'm describing, see https://cloud.google.com/generative-ai-app-builder/docs/parse-chunk-documents#parse-chunk-rag, mainly, a documentProcessingConfig field needs to be specifiable, as shown in https://cloud.google.com/generative-ai-app-builder/docs/parse-chunk-documents#example and https://cloud.google.com/generative-ai-app-builder/docs/parse-chunk-documents#turn-on-chunking

b/330174625

@dariowho
Copy link

Hi @rileykarson, I can take this one

@dariowho
Copy link

@etekin-amfam the PR for this issue does not include Layout Parsing and Chunking configuration, as those options were not available in either v1 or v1beta when this was implemented.

Those parameters are now available, I opened a separate issue to include them: #18390

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

4 participants