Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[eks] panic in cluster & fargate_profile hooks #2169

Open
vflaux opened this issue Sep 13, 2024 · 0 comments
Open

[eks] panic in cluster & fargate_profile hooks #2169

vflaux opened this issue Sep 13, 2024 · 0 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. service/eks Indicates issues or PRs that are related to eks-controller.

Comments

@vflaux
Copy link

vflaux commented Sep 13, 2024

Describe the bug
Panic in cluster & fargate_profile hooks.
Occurs when adpoting an eks cluster & fargate profile created recently.
Resolves by itself after some time.

There is the same dereferences in /github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster/hook.go:198 and /github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile/hook.go:79:

string(*desired.ko.Status.ACKResourceMetadata.ARN),

Apparently this is not set at some point.

Cluster stack 

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19ffb59]

goroutine 388 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x1c46100?, 0x34fbc20?})
	/usr/local/go/src/runtime/panic.go:770 +0x132
github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster.(*resourceManager).customUpdate(0xc000a7c1a0, {0x258bbe8, 0xc000ccde30}, 0xc00008ce68, 0xc00008d238, 0xc000dd0270)
	/github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster/hook.go:198 +0x20d9
github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster.(*resourceManager).sdkUpdate(...)
	/github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster/sdk.go:891
github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster.(*resourceManager).Update(0xc000a7c1a0, {0x258bbe8?, 0xc000ccde30?}, {0x2595488?, 0xc00008ce68?}, {0x2595488?, 0xc00008d238}, 0x5?)
	/github.com/aws-controllers-k8s/eks-controller/pkg/resource/cluster/manager.go:157 +0x78
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000985180, {0x258bbe8, 0xc000ccde30}, {0x2595420, 0xc000a7c1a0}, {0x2595488, 0xc00008ce68}, {0x2595488, 0xc00008d238})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:624 +0x3e3
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000985180, {0x258bbe8, 0xc000ccde30}, {0x2595420, 0xc000a7c1a0}, {0x2595488, 0xc00008ce38})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:367 +0x7de
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000985180, {0x258bbe8, 0xc000ccde30}, {0x2595420, 0xc000a7c1a0}, {0x2595488, 0xc00008ce38})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:303 +0x1cd
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000985180, {0x258bbe8, 0xc000ccd7d0}, {{{0xc000d78154?, 0x5?}, {0xc000d78150?, 0xc000a0ad10?}}})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:274 +0x7f5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x25904f0?, {0x258bbe8?, 0xc000ccd7d0?}, {{{0xc000d78154?, 0xb?}, {0xc000d78150?, 0x0?}}})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00002bcc0, {0x258bc20, 0xc000648e10}, {0x1d0f5c0, 0xc000c82e60})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00002bcc0, {0x258bc20, 0xc000648e10})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 176
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c
Fargate stack 
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1a0f84f]

goroutine 360 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x1c46100?, 0x34fbc20?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile.(*resourceManager).customUpdate(0xc00002e4e0, {0x258bbe8, 0xc00059e6c0}, 0xc000490290, 0xc000490408, 0xc000899d10)
    /github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile/hook.go:79 +0x2cf
github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile.(*resourceManager).sdkUpdate(...)
    /github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile/sdk.go:380
github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile.(*resourceManager).Update(0xc00002e4e0, {0x258bbe8?, 0xc00059e6c0?}, {0x2595558?, 0xc000490290?}, {0x2595558?, 0xc000490408}, 0x5?)
    /github.com/aws-controllers-k8s/eks-controller/pkg/resource/fargate_profile/manager.go:157 +0x78
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000651180, {0x258bbe8, 0xc00059e6c0}, {0x25954f0, 0xc00002e4e0}, {0x2595558, 0xc000490290}, {0x2595558, 0xc000490408})
    /go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:624 +0x3e3
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000651180, {0x258bbe8, 0xc00059e6c0}, {0x25954f0, 0xc00002e4e0}, {0x2595558, 0xc000490240})
    /go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:367 +0x7de
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000651180, {0x258bbe8, 0xc00059e6c0}, {0x25954f0, 0xc00002e4e0}, {0x2595558, 0xc000490240})
    /go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:303 +0x1cd
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000651180, {0x258bbe8, 0xc0005f1ce0}, {{{0xc00092d9d0?, 0x5?}, {0xc00092d9c0?, 0xc000dc7d10?}}})
    /go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.34.0/pkg/runtime/reconciler.go:274 +0x7f5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x25904f0?, {0x258bbe8?, 0xc0005f1ce0?}, {{{0xc00092d9d0?, 0xb?}, {0xc00092d9c0?, 0x0?}}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c055e0, {0x258bc20, 0xc0005903c0}, {0x1d0f5c0, 0xc000489980})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c055e0, {0x258bc20, 0xc0005903c0})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 176
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c
Stream closed EOF for kube-mgmt/aws-ack-eks-64f8845c6b-85tpp (controller)

Steps to reproduce
Create a new cluster & fargate profile.
Install ack eks and adopted resources for both cluster & fargate profile.

Expected outcome
A concise description of what you expected to happen.

Environment

  • Kubernetes version: 1.28
  • Using EKS (yes/no), if so version?: 1.28
  • AWS service targeted (S3, RDS, etc.): eks
@a-hilaly a-hilaly added kind/bug Categorizes issue or PR as related to a bug. service/eks Indicates issues or PRs that are related to eks-controller. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Sep 16, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. service/eks Indicates issues or PRs that are related to eks-controller.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@a-hilaly @vflaux