Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

track_total_hits is disabled for Elasticsearch Scroll API #6566

Closed
emmercm opened this issue Sep 26, 2024 · 1 comment
Closed

track_total_hits is disabled for Elasticsearch Scroll API #6566

emmercm opened this issue Sep 26, 2024 · 1 comment
Assignees

Comments

@emmercm
Copy link

emmercm commented Sep 26, 2024

Expected Behavior

Being allowed to batch reset workflows using a query string.

Actual Behavior

Given a query that returns workflows:

$ tctl --namespace workflow-engine-sandbox workflow list --query 'StartTime > "2024-09-26T12:30:00-07:00" AND CloseTime < "2024-09-26T12:40:00-07:00"' --pagesize 50
     WORKFLOW TYPE    |                WORKFLOW ID                 |                RUN ID                | TASK QUEUE | START TIME | EXECUTION TIME | END TIME  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:39:30Z | 34c93f70-0acd-4370-a605-7189cf16b9b2 | SANDBOX    | 19:39:30   | 19:39:30       | 19:39:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:39:00Z | 0bc4a9d0-41f6-48f1-ae92-66621255ca89 | SANDBOX    | 19:39:00   | 19:39:00       | 19:39:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:38:30Z | a10182da-b260-4e61-b846-8a49907785a3 | SANDBOX    | 19:38:30   | 19:38:30       | 19:38:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:38:00Z | 7c7abbde-4290-4838-89f3-063ed7609b64 | SANDBOX    | 19:38:00   | 19:38:00       | 19:38:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:37:30Z | 78e4bd13-8e1e-4d98-b8e8-d0d966985f47 | SANDBOX    | 19:37:30   | 19:37:30       | 19:37:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:37:00Z | 3fa2bad1-eec9-4403-a3a8-016471594550 | SANDBOX    | 19:37:00   | 19:37:00       | 19:37:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:36:30Z | 54eac827-556f-41fd-874c-3ab1db80a1d3 | SANDBOX    | 19:36:30   | 19:36:30       | 19:36:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:36:00Z | 10839b2b-cece-4060-979a-26b178f2c317 | SANDBOX    | 19:36:00   | 19:36:00       | 19:36:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:35:30Z | 5e252f0e-d24b-4149-b91e-2d09bd6dc74b | SANDBOX    | 19:35:30   | 19:35:30       | 19:35:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:35:00Z | 24d9f674-606b-461d-bf35-d9eb29c52d5b | SANDBOX    | 19:35:00   | 19:35:00       | 19:35:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:34:30Z | bf093800-3dcf-4e5c-a2bc-4426336d50d0 | SANDBOX    | 19:34:30   | 19:34:30       | 19:34:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:34:00Z | 3e5f84fc-5833-4588-a56a-68c1c6ee4c14 | SANDBOX    | 19:34:00   | 19:34:00       | 19:34:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:33:30Z | 1023d2d5-f60c-4ca4-8a2d-b5f2d19cb849 | SANDBOX    | 19:33:30   | 19:33:30       | 19:33:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:33:00Z | 78ca18ba-0023-4523-a818-5c494c0c94f3 | SANDBOX    | 19:33:00   | 19:33:00       | 19:33:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:32:30Z | 894583ee-b3e5-4552-94f9-d0ccdfcd32c6 | SANDBOX    | 19:32:30   | 19:32:30       | 19:32:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:32:00Z | ee0e407d-260b-44aa-9e38-4bae69fb4717 | SANDBOX    | 19:32:00   | 19:32:00       | 19:32:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:31:30Z | 0385402d-abdb-49fd-b64e-0ccde8d576aa | SANDBOX    | 19:31:30   | 19:31:30       | 19:31:30  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:31:00Z | f1e1f7d5-632b-4e32-b2bc-be4c04abc7ef | SANDBOX    | 19:31:00   | 19:31:00       | 19:31:00  
  HealthcheckWorkflow | 30-second-healthcheck-2024-09-26T19:30:30Z | 4533ee7f-7608-4548-b853-a7e989882bce | SANDBOX    | 19:30:30   | 19:30:30       | 19:30:30

when I try to batch reset those workflows, I get the error:

$ tctl --namespace workflow-engine-sandbox workflow reset-batch --query 'StartTime > "2024-09-26T12:30:00-07:00" AND CloseTime < "2024-09-26T12:40:00-07:00"' --reason "test" --reset_type FirstWorkflowTask --dry_run
num of excludes: 0
Error: Failed to list workflow.
Error Details: ScanWorkflowExecutions failed: elastic: Error 400 (Bad Request): Validation Failed: 1: disabling [track_total_hits] is not allowed in a scroll context; [type=action_request_validation_exception]
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)

omitting the --dry_run makes no difference.

I understand that the setting cannot be disabled for the Elasticsearch Scroll API, and I think this is the exact problematic line: 3649fcf#diff-d50f473f260834fc429ccfa7f37b189b51b94bd82de0c1b75c7db2233e940f66R196

Steps to Reproduce the Problem

I'm fairly sure this broke for everyone starting with v1.24.0.

Specifications

  • Platform: Docker
  • Server version: temporalio/server:1.24.2
  • tctl version (provided by the Docker image): v1.18.0
  • Persistence store: MySQL
  • Advanced visibility store: Elasticsearch v7.10
@rodrigozhou rodrigozhou self-assigned this Oct 2, 2024
rodrigozhou added a commit that referenced this issue Oct 2, 2024
## What changed?
<!-- Describe what has changed in this PR -->
Remove track_total_hits from Scroll API

## Why?
<!-- Tell your future self why have you made these changes -->
`track_total_hits` cannot be disable in Scroll API.

Issue: #6566

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
@rodrigozhou
Copy link
Contributor

The issue is fixed in #6592, and it should be included in the next patch release.

dnr pushed a commit that referenced this issue Oct 4, 2024
## What changed?
<!-- Describe what has changed in this PR -->
Remove track_total_hits from Scroll API

## Why?
<!-- Tell your future self why have you made these changes -->
`track_total_hits` cannot be disable in Scroll API.

Issue: #6566

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
dnr pushed a commit that referenced this issue Oct 11, 2024
## What changed?
<!-- Describe what has changed in this PR -->
Remove track_total_hits from Scroll API

## Why?
<!-- Tell your future self why have you made these changes -->
`track_total_hits` cannot be disable in Scroll API.

Issue: #6566

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

2 participants