enhance: Add json key inverted index in stats for optimization #38039

JsDove · 2024-11-27T01:52:07Z

Add json key inverted index in stats for optimization
#36995

mergify · 2024-11-27T01:52:46Z

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

Title Format: The PR title must begin with one of these prefixes:

feat: for introducing a new feature.
fix: for bug fixes.
enhance: for improvements to existing functionality.
test: for add tests to existing functionality.
doc: for modifying documentation.
auto: for the pull request from bot.

Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly

Please review and update your PR to comply with these guidelines.

mergify · 2024-11-27T02:40:11Z

@JsDove cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-27T02:51:11Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-27T02:58:13Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-27T04:11:57Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-02T04:46:45Z

@JsDove cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-02T05:59:30Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-03T02:32:45Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-03T02:32:46Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-03T02:38:11Z

@JsDove cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-10T01:58:39Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-10T01:58:39Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-10T01:58:39Z

@JsDove cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-10T04:04:43Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-10T07:02:04Z

@JsDove cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-13T09:11:41Z

@JsDove cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-13T09:20:52Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-02-14T04:11:22Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

…39876) Add json key inverted index in stats for optimization issue: #36995 pr: #38039 --------- Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com>

mergify · 2025-02-17T12:57:17Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-02-18T09:58:51Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

JsDove · 2025-02-18T11:20:34Z

rerun go-sdk

mergify · 2025-02-18T12:08:19Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-02-18T15:36:45Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

…9972) fix: exists expr on nested dictionaries fail in JSONkeyIndex issue: #39963 pr: #38039 Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

enhance: compress jsonstatslog path only save filename issue: #36995 pr: #38039 Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> jsoncontainexpr unaryexpr binaryexpr json index optimization Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> improve jsonContainSExpr Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> add jsonindex test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> support load json index after loadsegment Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> improve statschecker unittest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> jsonindex expr code format Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix go format Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix controllerbasetest fail Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonindex memeroy leak Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonkey go format Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonindex go codeformat Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> improve jsoninvert unitest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> delete unuse code Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> refine test_json_key_index Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix cpp unitest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> delete loginfo Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix complie error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix codeformat Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix createindex again hang up Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix unitest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix createindex hang Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix triggerstatstask go ut Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonindex filter error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonindex filter error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix format Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> improve jsonkey unitest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> remove sealsegment chunknum assert Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix codereview Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonindex parse string to int Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix unitest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> improve Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix go test error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> improve Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> adjust jsonkey tantivy buffer size Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix: nested jsonarray cause incorrect index Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix: nested jsonarray cause incorrect index Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> feat growingsegment jsonkey index Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> feat growingsegment jsonkey index Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix complie error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix complie error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix complie error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix complie error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonkeyindex array contain Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonkeyindex arrayindex filter error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonkeyindex arrayindex filter error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> format Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> adjust createjsonindex trigger time Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> Revert "adjust createjsonindex trigger time" This reverts commit 84ea08a. fix jsoncontain Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsoncontain Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> growing jsonkey index no support offsetinput Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix yaml format Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> add jsonkeyindex unitest Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix e2e jsoncontain Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> bug fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> bug fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix go ut Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix ut Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> growing json key fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> remova skip Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> move upper to for {} outside Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> crashfix: concurrentvector json use view_element Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> rename variable Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> add json key stats enabled in yaml Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix complie error Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> modiy Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix jsonkey index read-write confict Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> revert jobmanager Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> revert Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> add growingjson index switch Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix codecheck Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix code check Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix lock Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix growing bitset out of size Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> remove mutx Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix gosdk test Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> replace interface Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> modify json key tantivy memory size Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

mergify · 2025-02-19T09:43:24Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

JsDove · 2025-02-19T11:32:13Z

rerun go-sdk

mergify · 2025-02-19T12:21:10Z

@JsDove E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-02-19T12:23:18Z

@JsDove go-sdk check failed, comment rerun go-sdk can trigger the job again.

pr: #38039 Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Replace the position of EnabledJSONKeyStats issue: #36995 pr: #38039 --------- Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Nov 27, 2024

sre-ci-robot requested review from bigsheeper and cydrain November 27, 2024 01:52

sre-ci-robot added area/compilation area/internal-api labels Nov 27, 2024

mergify bot added the dco-passed DCO check passed. label Nov 27, 2024

mergify bot added the do-not-merge/invalid-pr-format label Nov 27, 2024

JsDove changed the title ~~Enhancement: Add json key inverted index in stats for optimization~~ enhance: Add json key inverted index in stats for optimization Nov 27, 2024

mergify bot added kind/enhancement Issues or changes related to enhancement and removed do-not-merge/invalid-pr-format labels Nov 27, 2024

JsDove force-pushed the optimization_json branch from a5befae to e1531e7 Compare December 10, 2024 04:00

czs007 mentioned this pull request Dec 11, 2024

add json key inverted index in stats to speed up json expr #36996

Closed

JsDove force-pushed the optimization_json branch from 8f74146 to 2910f86 Compare December 13, 2024 06:34

JsDove force-pushed the optimization_json branch 2 times, most recently from 0a41326 to 2c09b04 Compare February 17, 2025 08:06

JsDove mentioned this pull request Feb 17, 2025

enhance: [2.5]Add scalarStats mmap to the Milvus configuration items #39945

Closed

JsDove mentioned this pull request Feb 18, 2025

fix: [2.5]exists expr on nested dictionaries fail in JSONkeyIndex #39972

Merged

JsDove force-pushed the optimization_json branch 2 times, most recently from 58432f2 to ff88d55 Compare February 18, 2025 09:36

JsDove mentioned this pull request Feb 18, 2025

enhance: [2.5]compress jsonstatslog path only save filename #39979

Merged

sre-ci-robot pushed a commit that referenced this pull request Feb 19, 2025

enhance: [2.5]compress jsonstatslog path only save filename (#39979)

04175d8

enhance: compress jsonstatslog path only save filename issue: #36995 pr: #38039 Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

luzhang and others added 4 commits February 19, 2025 16:10

support strong query

8c3a323

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

fix: exists expr on nested dictionaries fail in JSONkeyIndex

12267ad

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

compress jsonstatslog path only save filename

fa612ac

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> modify json key tantivy memory size Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com> fix Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

JsDove force-pushed the optimization_json branch from ff88d55 to fa612ac Compare February 19, 2025 08:10

czs007 mentioned this pull request Feb 19, 2025

fix: Fix wrong trigger of json key index #40026

Merged

czs007 added a commit that referenced this pull request Feb 19, 2025

fix: Fix wrong trigger of json key index (#40026)

f142b5b

pr: #38039 Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

This was referenced Feb 20, 2025

fix: [2.5]fix the garbage collection cleanup logic of jsonkeyindex #40039

Open

fix: [2.5]Replace the position of EnabledJSONKeyStats #40108

Merged

sre-ci-robot pushed a commit that referenced this pull request Feb 22, 2025

fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108)

c1de61f

Replace the position of EnabledJSONKeyStats issue: #36995 pr: #38039 --------- Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: Add json key inverted index in stats for optimization #38039

enhance: Add json key inverted index in stats for optimization #38039

JsDove commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Dec 2, 2024

mergify bot commented Dec 2, 2024

mergify bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Feb 14, 2025

mergify bot commented Feb 17, 2025

mergify bot commented Feb 18, 2025

JsDove commented Feb 18, 2025

mergify bot commented Feb 18, 2025

mergify bot commented Feb 18, 2025

mergify bot commented Feb 19, 2025

JsDove commented Feb 19, 2025

mergify bot commented Feb 19, 2025

mergify bot commented Feb 19, 2025

enhance: Add json key inverted index in stats for optimization #38039

Are you sure you want to change the base?

enhance: Add json key inverted index in stats for optimization #38039

Conversation

JsDove commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Dec 2, 2024

mergify bot commented Dec 2, 2024

mergify bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 10, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Feb 14, 2025

mergify bot commented Feb 17, 2025

mergify bot commented Feb 18, 2025

JsDove commented Feb 18, 2025

mergify bot commented Feb 18, 2025

mergify bot commented Feb 18, 2025

mergify bot commented Feb 19, 2025

JsDove commented Feb 19, 2025

mergify bot commented Feb 19, 2025

mergify bot commented Feb 19, 2025