Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

fix: Prevent crash due to concurrent map access in Go plugin log processing #1284

Merged
merged 5 commits into from
Dec 29, 2023

Conversation

quzard
Copy link
Collaborator

@quzard quzard commented Dec 26, 2023

Issue Description

A fatal error, "concurrent map read and map write," occurs when using Go plugins under high log volume and concurrent conditions. The root cause of this issue is traced back to the ReceiveLogGroup function in plugin_runner_v1.go. The context passed to subsequent log processing did not undergo a deep copy, leading to the use of the same map object.

Problematic Scenario

In the aforementioned function, while the context map is being assigned new values, the Add function in aggregator_context.go is concurrently accessing values from the same context. This simultaneous operation on the same map object results in concurrent read and write errors, which is not permissible in Go without proper synchronization mechanisms.

Resolution

To address this problem, a deep copy of the context map is now implemented for each log processing instance. By doing so, each context uses a separate map object, eliminating the reuse of the same map and thereby preventing concurrent read and write conflicts. This deep-copy approach ensures that each log processor operates on an independent copy of the context, maintaining the integrity of the map access throughout the logging system.

pluginmanager/plugin_runner_v1.go Outdated Show resolved Hide resolved
@quzard quzard changed the title fix: The map is shared, which leads to concurrent read and write issues when reading context in aggregator_context.go:Add and writing context in plugin_runner_v1.go:ReceiveLogGroup fix: Enhance the context isolation in Go plugin log processing to resolve concurrent mapping access errors. Dec 28, 2023
@yyuuttaaoo yyuuttaaoo changed the title fix: Enhance the context isolation in Go plugin log processing to resolve concurrent mapping access errors. fix: Prevent crash due to concurrent map access in Go plugin log processing Dec 29, 2023
@yyuuttaaoo yyuuttaaoo added the bug Something isn't working label Dec 29, 2023
@yyuuttaaoo yyuuttaaoo added this to the v2.0 milestone Dec 29, 2023
@yyuuttaaoo yyuuttaaoo merged commit 6ba9dce into alibaba:main Dec 29, 2023
17 checks passed
@quzard quzard deleted the fix-context branch February 28, 2024 02:25
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants