Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

CI Failure (segment replaced with no change in size) in TieredStorageTest.test_tiered_storage #13753

Closed
abhijat opened this issue Sep 28, 2023 · 4 comments · Fixed by #13896
Closed

CI Failure (segment replaced with no change in size) in TieredStorageTest.test_tiered_storage #13753

abhijat opened this issue Sep 28, 2023 · 4 comments · Fixed by #13896
Assignees
@VladLazar
Labels
area/cloud-storage Shadow indexing subsystem ci-failure kind/bug Something isn't working sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low.

Comments

@abhijat
Copy link
Contributor

abhijat commented Sep 28, 2023

https://buildkite.com/redpanda/redpanda/builds/37795

Module: rptest.tests.tiered_storage_model_test
Class: TieredStorageTest
Method: test_tiered_storage
Arguments: {
    "test_case": {
        "name": "(TS_Read == True, SpilloverManifestUploaded == True)"
    },
    "cloud_storage_type": 2
}

test_id:    TieredStorageTest.test_tiered_storage
status:     FAIL
run time:   108.422 seconds

<BadLogLines nodes=docker-rp-19(8),docker-rp-11(8),docker-rp-12(8) example="ERROR 2023-09-28 03:52:28,062 [shard 1:main] cloud_storage - partition_manifest.cc:1024 - [{kafka/topic-vfoseiytae/0}] New replacement segment has the same size as replaced segment: new_segment: {o=29720-30744 t={timestamp: 1695873131347}-{timestamp: 1695873142470}}, replaced_segment: {o=29720-30744 t={timestamp: 1695873131347}-{timestamp: 1695873142470}}">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1244, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-19(8),docker-rp-11(8),docker-rp-12(8) example="ERROR 2023-09-28 03:52:28,062 [shard 1:main] cloud_storage - partition_manifest.cc:1024 - [{kafka/topic-vfoseiytae/0}] New replacement segment has the same size as replaced segment: new_segment: {o=29720-30744 t={timestamp: 1695873131347}-{timestamp: 1695873142470}}, replaced_segment: {o=29720-30744 t={timestamp: 1695873131347}-{timestamp: 1695873142470}}">
@abhijat abhijat added kind/bug Something isn't working ci-failure area/cloud-storage Shadow indexing subsystem labels Sep 28, 2023
@abhijat
Copy link
Contributor Author

abhijat commented Sep 28, 2023 •
edited
Loading

Also seen on:

https://buildkite.com/redpanda/redpanda/builds/37810

Module: rptest.tests.cloud_storage_usage_test
Class:  CloudStorageUsageTest
Method: test_cloud_storage_usage_reporting_with_partition_moves

====================================================================================================
test_id:    rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves
status:     FAIL
run time:   1 minute 28.912 seconds


    <BadLogLines nodes=docker-rp-20(4) example="ERROR 2023-09-28 06:54:56,162 [shard 1:main] cloud_storage - partition_manifest.cc:1024 - [{kafka/test-topic-2/0}] New replacement segment has the same size as replaced segment: new_segment: {o=347-382 t={timestamp: 1695884063937}-{timestamp: 1695884065937}}, replaced_segment: {o=347-382 t={timestamp: 1695884063937}-{timestamp: 1695884065875}}">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1244, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-20(4) example="ERROR 2023-09-28 06:54:56,162 [shard 1:main] cloud_storage - partition_manifest.cc:1024 - [{kafka/test-topic-2/0}] New replacement segment has the same size as replaced segment: new_segment: {o=347-382 t={timestamp: 1695884063937}-{timestamp: 1695884065937}}, replaced_segment: {o=347-382 t={timestamp: 1695884063937}-{timestamp: 1695884065875}}">

@Lazin Lazin added the sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low. label Sep 28, 2023
@Lazin Lazin self-assigned this Sep 28, 2023
@VladLazar
Copy link
Contributor

Sorry, Evgeny. Hadn't seen you self assigned.

Here's what happens:
Upload loop picks 4 candidates, uploads to cloud storage and replicates a batch with 4 new segments. After the replication command succeeds, the topic retention is updated. This re-starts the archiver. When the archiver restarts it picks the same 4 segments and replicates the addition commands again. Finally, the partition manifest detects this, refuses the commands and prints the error log.

TODO: add logs

@VladLazar VladLazar assigned VladLazar and unassigned Lazin Sep 28, 2023
@abhijat
Copy link
Contributor Author

abhijat commented Sep 29, 2023

Sorry, Evgeny. Hadn't seen you self assigned.

Here's what happens: Upload loop picks 4 candidates, uploads to cloud storage and replicates a batch with 4 new segments. After the replication command succeeds, the topic retention is updated. This re-starts the archiver. When the archiver restarts it picks the same 4 segments and replicates the addition commands again. Finally, the partition manifest detects this, refuses the commands and prints the error log.

TODO: add logs

This sounds pretty similar to #12846

@VladLazar
Copy link
Contributor

We should fix this. In this specific case, we handle it gracefully, but there's no guarantee we'll do this for all commands.

@redpanda-data redpanda-data deleted a comment from abhijat Sep 29, 2023
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 3, 2023
archival: handle same term syncs in archival STM
310ad4e 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 3, 2023
archival: handle same term syncs in archival STM
47f9e70 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
This was referenced Oct 3, 2023
Merged
Merged
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 3, 2023
archival: handle same term syncs in archival STM
9d1903d 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 5, 2023
archival: handle same term syncs in archival STM
15f69c8 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 5, 2023
archival: handle same term syncs in archival STM
ada37dd 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
@andijcr andijcr mentioned this issue Oct 6, 2023
Closed
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 10, 2023
archival: handle same term syncs in archival STM
c297b3f 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 11, 2023
archival: handle same term syncs in archival STM
636b16e 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 11, 2023
archival: handle same term syncs in archival STM
78fb379 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
use the cached last offset of the last replicated batch and wait on its
application.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 12, 2023
archival: handle same term syncs in archival STM
f55b6bd 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 12, 2023
archival: handle same term syncs in archival STM
2076c36 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 12, 2023
archival: handle same term syncs in archival STM
2a10fc8 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 13, 2023
archival: handle same term syncs in archival STM
8b66d00 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 16, 2023
archival: handle same term syncs in archival STM
efb20b7 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 20, 2023
archival: handle same term syncs in archival STM
151a215 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 20, 2023
archival: handle same term syncs in archival STM
2a48c5e 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Oct 20, 2023
archival: handle same term syncs in archival STM
0473025 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes redpanda-data#13753
VladLazar pushed a commit that referenced this issue Nov 1, 2023
archival: handle same term syncs in archival STM
f0eeb41 
Previously, the archival STM used `persisted_stm::sync` in order to
ensure that its log is up to date before issuing new commands. This
works fine for leadership changes since they imply a term change.

However, the archiver may need to sync during the same term. We've seen
scenarios like this:

> Upload loop picks 4 candidates, uploads to cloud storage and
replicates a batch with 4 new segments. After the replication command
succeeds, the topic retention is updated. This re-starts the archiver.
When the archiver restarts it picks the same 4 segments and replicates
the addition commands again. Finally, the partition manifest detects
this, refuses the commands and prints the error log.

The previous sync method does not support such cases sync they happen in
the same term. To fix this, the archival STM now implements its own sync
method. If it detects that the term has changed, it will maintain the
pre-existing behaviour: sync up to the latest term. Otherwise, it will
wait for any ongoing replications to finish and then wait for the
commited offest to be applied.

Fixes #13753
@github-actions github-actions bot mentioned this issue Nov 13, 2023
Merged
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@VladLazar VladLazar

Labels
area/cloud-storage Shadow indexing subsystem ci-failure kind/bug Something isn't working sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

archival: same term sync-ing VladLazar/redpanda
3 participants
@Lazin @abhijat @VladLazar