Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Drop task on serialization error #3803

Merged
merged 5 commits into from
Jan 12, 2023

Conversation

yycptt
Copy link
Member

@yycptt yycptt commented Jan 12, 2023

What changed?

  • Drop task on unknown encoding and blob deserialization error, which is likely due to data corruption.
  • Remove panic in NewDataBlob when encoding is invalid. Upon decoding, unknown encoding error will be returned.

Why?

How did you test it?

Potential risks

Is hotfix candidate?
Yes

@yycptt yycptt marked this pull request as ready for review January 12, 2023 00:57
@yycptt yycptt requested a review from a team as a code owner January 12, 2023 00:57
@yycptt yycptt requested review from yiminc and meiliang86 January 12, 2023 00:58
@yycptt yycptt force-pushed the drop-task-on-serialization-error branch from 43f676b to 414e188 Compare January 12, 2023 01:41
Comment on lines 117 to 125
DeserializationError struct {
msg string
encodingType enumspb.EncodingType
wrappedErr error
}

// UnknownEncodingTypeError is an error type for unknown or unsupported encoding type
UnknownEncodingTypeError struct {
encodingType enumspb.EncodingType
encodingTypeStr string
supportedEncodingStr []string
Copy link
Member Author

@yycptt yycptt Jan 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should setup an alert to catch those error types returned from persistence client and the alert should fire whenever those errors occur. Otherwise some components may encounter those errors and retry slowly, which will never trigger the persistence error alert.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@yycptt yycptt added the release/1.19.1 Patches for v1.19.1 label Jan 12, 2023
Comment on lines 117 to 125
DeserializationError struct {
msg string
encodingType enumspb.EncodingType
wrappedErr error
}

// UnknownEncodingTypeError is an error type for unknown or unsupported encoding type
UnknownEncodingTypeError struct {
encodingType enumspb.EncodingType
encodingTypeStr string
supportedEncodingStr []string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

common/persistence/serialization/serializer.go Outdated Show resolved Hide resolved
service/history/queues/executable.go Outdated Show resolved Hide resolved
@yycptt yycptt force-pushed the drop-task-on-serialization-error branch from 649eb5c to fea053e Compare January 12, 2023 22:40
@yycptt yycptt merged commit 26503cf into temporalio:master Jan 12, 2023
@yycptt yycptt deleted the drop-task-on-serialization-error branch January 12, 2023 23:20
yycptt added a commit that referenced this pull request Jan 12, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
release/1.19.1 Patches for v1.19.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants