Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

If all records fail suggest_truncate is never sent to the source #2825

Closed
imotov opened this issue Feb 16, 2023 · 0 comments · Fixed by #2826
Closed

If all records fail suggest_truncate is never sent to the source #2825

imotov opened this issue Feb 16, 2023 · 0 comments · Fixed by #2826
Labels
bug Something isn't working

Comments

@imotov
Copy link
Collaborator

imotov commented Feb 16, 2023

Describe the bug
In my testing for #2699 implementation I noticed that when all documents fail suggest_truncate is never sent to the SourceActor, as a result queue is never notified that it can clean older records. In my experience this situation happens more often than one could expect. For example, a new version of software is installed and it starts generating logs in a slightly different format or mappings are fat-fingered making all login messages suddenly fail. Without proper monitoring this situation can remain unnoticed for quite a while, sometimes days until users wonder why they cannot see new log messages in the system.

Steps to reproduce (if applicable)

  1. Observe SourceActor's suggest_truncate behavior while indexing all failed messages.

Expected behavior
The suggest_truncate should be sent periodically even if all documents happened to fail.
@imotov imotov added the bug Something isn't working label Feb 16, 2023
imotov added a commit to imotov/quickwit that referenced this issue Feb 16, 2023
@imotov
Refactor Indexer to talk directly to Publisher
5d4716f 
Ensures that publisher is still notified even if all documents fail.

Fixes quickwit-oss#2825
@imotov imotov mentioned this issue Feb 16, 2023
Merged
imotov added a commit to imotov/quickwit that referenced this issue Mar 14, 2023
@imotov
Add an option to force and wait for commit
57d3f90 
Adds an option to force a commit or wait for the next commit while
issuing an ingest command. This commit only implements it for the
ingest endpoint. If all records in the batch fail, the command will
hang until at least one good document is indexed until quickwit-oss#2825 is merged
in or the document parsing is moved to the early stages.

See quickwit-oss#2699
trinity-1686a added a commit that referenced this issue Mar 17, 2023
@imotov @trinity-1686a
Add an option to force or wait for commit (#2995)
26927f0 
* Add an option to force and wait for commit

Adds an option to force a commit or wait for the next commit while
issuing an ingest command. This commit only implements it for the
ingest endpoint. If all records in the batch fail, the command will
hang until at least one good document is indexed until #2825 is merged
in or the document parsing is moved to the early stages.

See #2699

* Add comments to the notification triggering logic

* Fix rest tests

* Remove extra current_offset setting

---------

Co-authored-by: trinity-1686a <trinity@quickwit.io>
imotov added a commit that referenced this issue Mar 21, 2023
@imotov @guilload
Refactor Indexer to talk directly to Publisher (#2826)
a119c33 
Ensures that publisher is still notified even if all documents in the batch fail.

Fixes #2825

Co-authored-by: Adrien Guillo <adrien.guillo@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Refactor Indexer to talk directly to Publisher imotov/quickwit
1 participant
@imotov