Virtual Kubelet pod crashes very frequently

[Sharathmk99]

What happened:

Our Virtual Kubelet pod crashes and restarts very frequently. During the restart Node goes to NotReady status and it can interrupt scheduling.

Error,

goroutine 17104 [select]:
golang.org/x/time/rate.(*Limiter).wait(0xc0008d27d0, {0x2803760, 0x39d2680}, 0x1, {0x1d5fca0?, 0x0?, 0x39a0e80?}, 0x235c9f8)
	/go/pkg/mod/golang.org/x/time@v0.3.0/rate/rate.go:283 +0x3e5
golang.org/x/time/rate.(*Limiter).WaitN(0x0?, {0x2803760, 0x39d2680}, 0x49484f?)
	/go/pkg/mod/golang.org/x/time@v0.3.0/rate/rate.go:246 +0x50
golang.org/x/time/rate.(*Limiter).Wait(...)
	/go/pkg/mod/golang.org/x/time@v0.3.0/rate/rate.go:231
k8s.io/client-go/util/flowcontrol.(*tokenBucketRateLimiter).Wait(0x1d5fca0?, {0x2803760?, 0x39d2680?})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/util/flowcontrol/throttle.go:131 +0x25
k8s.io/client-go/rest.(*Request).tryThrottleWithInfo(0xc0101ea480, {0x2803760, 0x39d2680}, {0x0, 0x0})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/rest/request.go:614 +0xb4
k8s.io/client-go/rest.(*Request).tryThrottle(...)
	/go/pkg/mod/k8s.io/client-go@v0.28.3/rest/request.go:642
k8s.io/client-go/rest.(*Request).request(0xc0101ea480, {0x2803760, 0x39d2680}, 0x2?)
	/go/pkg/mod/k8s.io/client-go@v0.28.3/rest/request.go:989 +0x232
k8s.io/client-go/rest.(*Request).Do(0xc0101ea480, {0x2803760, 0x39d2680})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/rest/request.go:1063 +0xb0
k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List(0xc0082930c0, {0x2803760, 0x39d2680}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/kubernetes/typed/core/v1/pod.go:99 +0x165
k8s.io/client-go/informers/core/v1.NewFilteredPodInformer.func1({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/informers/core/v1/pod.go:65 +0x149
k8s.io/client-go/tools/cache.(*ListWatch).List(0xc011440528?, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/listwatch.go:106 +0x3c
k8s.io/client-go/tools/cache.(*Reflector).list.func1.2({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:488 +0x55
k8s.io/client-go/tools/cache.(*Reflector).list.func1.SimplePageFunc.func3({0x27eca18?, 0x21dfd40?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...})
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/pager/pager.go:40 +0x3d
k8s.io/client-go/tools/pager.(*ListPager).list(0xc012011f20, {0x2803648, 0x39d2680}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/pager/pager.go:108 +0x166
k8s.io/client-go/tools/pager.(*ListPager).ListWithAlloc(...)
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/pager/pager.go:89
k8s.io/client-go/tools/cache.(*Reflector).list.func1()
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:513 +0x1dc
created by k8s.io/client-go/tools/cache.(*Reflector).list in goroutine 17103
	/go/pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:479 +0x38b

goroutine 18627 [runnable]:
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/manager.(*manager).StartNamespace.func2()
	/tmp/builder/pkg/virtualKubelet/reflection/manager/manager.go:185
created by github.com/liqotech/liqo/pkg/virtualKubelet/reflection/manager.(*manager).StartNamespace in goroutine 146
	/tmp/builder/pkg/virtualKubelet/reflection/manager/manager.go:185 +0xaf7

What you expected to happen:

Virtual Kubelet pod doesn't restart

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Liqo version: v0.10.1
• Liqoctl version: v0.10.1
• Kubernetes version (use kubectl version): 1.27.x
• Cloud provider or hardware configuration: on-premises
• Node image:
• Network plugin and version:
• Install tools:
• Others: