Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Update approximate_threshold to 15K documents #2229

Merged
merged 4 commits into from
Oct 23, 2024

Conversation

VijayanB
Copy link
Member

@VijayanB VijayanB commented Oct 22, 2024

Description

After comparing indexing and search performance, we are updating default value to be 15000.
Updated bwc rolling upgrade test and restart test to reset threshold to 0 to build graph always. Few tests checks
for graph in cache and having 15K will break that test. Hence, when new cluster comes up, we will update the setting to 0, force merge if require, before doing any new indexing.

Related Issues

#1942

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@VijayanB VijayanB changed the title Update threshold to 15K documents Update approximate_threshold to 15K documents Oct 22, 2024
@VijayanB VijayanB marked this pull request as draft October 22, 2024 07:17
After comparing indexing and search performance, we are updating
default value to be 15000.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
@VijayanB VijayanB force-pushed the update-threshold branch 8 times, most recently from 284624e to dbf499e Compare October 23, 2024 17:01
@VijayanB VijayanB marked this pull request as ready for review October 23, 2024 17:47
@VijayanB VijayanB self-assigned this Oct 23, 2024
Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
@VijayanB VijayanB merged commit eb0a3c7 into opensearch-project:main Oct 23, 2024
30 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-2229-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 eb0a3c7454cb33346f135161beb21f46f43b8457
# Push it to GitHub
git push --set-upstream origin backport/backport-2229-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2229-to-2.x.

VijayanB added a commit to VijayanB/k-NN-2 that referenced this pull request Oct 23, 2024
* Update threshold to 15K documents

After comparing indexing and search performance, we are updating
default value to be 15000.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Fix bwc test

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Update test method

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Flush data after index

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
VijayanB added a commit that referenced this pull request Oct 24, 2024
…2231)

* Update approximate_threshold to 15K documents (#2229)

* Update threshold to 15K documents

After comparing indexing and search performance, we are updating
default value to be 15000.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Fix bwc test

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Update test method

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Flush data after index

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Remove udpate cluster setting

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* update warmup rolling upgrade scenario

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 24, 2024
…2231)

* Update approximate_threshold to 15K documents (#2229)

* Update threshold to 15K documents

After comparing indexing and search performance, we are updating
default value to be 15000.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Fix bwc test

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Update test method

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Flush data after index

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Remove udpate cluster setting

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* update warmup rolling upgrade scenario

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
(cherry picked from commit bdbddcf)
VijayanB added a commit that referenced this pull request Oct 24, 2024
…2231) (#2234)

* Update approximate_threshold to 15K documents (#2229)

* Update threshold to 15K documents

After comparing indexing and search performance, we are updating
default value to be 15000.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Fix bwc test

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Update test method

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Flush data after index

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Remove udpate cluster setting

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* update warmup rolling upgrade scenario

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
(cherry picked from commit bdbddcf)

Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants