[BUG]The yurthub consuming too much CPU

[huweihuang]

What happened:

The yurthub consuming too much CPU, If there is no CPU limit, it will consume more than 4 cores. Not normal compared to other nodes

Environment:

• OpenYurt version:

yurthub image id

docker.io/openyurt/yurthub                                  latest                 19f5b9b7fb1dc       19.6MB

Anything else we need to know?:

top CPU of pprof:

(pprof) top 20
Showing nodes accounting for 52.09s, 91.56% of 56.89s total
Dropped 50 nodes (cum <= 0.28s)
Showing top 20 nodes out of 81
      flat  flat%   sum%        cum   cum%
    17.53s 30.81% 30.81%     19.12s 33.61%  syscall.Syscall
    11.85s 20.83% 51.64%     11.85s 20.83%  runtime.epollwait
    10.56s 18.56% 70.21%     10.56s 18.56%  runtime.futex
     1.93s  3.39% 73.60%      2.05s  3.60%  runtime.casgstatus
     1.24s  2.18% 75.78%     32.19s 56.58%  runtime.findrunnable
     1.22s  2.14% 77.92%      1.22s  2.14%  runtime.(*randomEnum).next
     0.97s  1.71% 79.63%      1.83s  3.22%  runtime.lock2
     0.84s  1.48% 81.10%      0.84s  1.48%  runtime.nanotime (inline)
     0.75s  1.32% 82.42%      3.89s  6.84%  runtime.stealWork
     0.69s  1.21% 83.64%      1.33s  2.34%  runtime.checkTimers
     0.69s  1.21% 84.85%      0.69s  1.21%  runtime.runqgrab
     0.61s  1.07% 85.92%      0.61s  1.07%  runtime.procyield
     0.54s  0.95% 86.87%     13.01s 22.87%  runtime.netpoll
     0.54s  0.95% 87.82%      0.54s  0.95%  runtime/internal/atomic.(*Uintptr).CompareAndSwap (inline)
     0.46s  0.81% 88.63%     21.09s 37.07%  internal/poll.(*FD).Write
     0.41s  0.72% 89.35%      0.94s  1.65%  runtime.netpollblock
     0.37s  0.65% 90.00%      0.37s  0.65%  runtime.netpollblockcommit
     0.33s  0.58% 90.58%      0.33s  0.58%  runtime.pMask.read
     0.32s  0.56% 91.14%      0.32s  0.56%  runtime.pidleput
     0.24s  0.42% 91.56%      1.36s  2.39%  internal/poll.(*pollDesc).wait

flame graph

goroutine:

File: yurthub
Type: goroutine
Time: Jul 11, 2023 at 3:28pm (WIB)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 186, 100% of 186 total
Showing top 10 nodes out of 195
      flat  flat%   sum%        cum   cum%
       182 97.85% 97.85%        182 97.85%  runtime.gopark
         2  1.08% 98.92%          2  1.08%  syscall.Syscall
         1  0.54% 99.46%          1  0.54%  runtime.notetsleepg
         1  0.54%   100%          1  0.54%  runtime/pprof.runtime_goroutineProfileWithLabels
         0     0%   100%          1  0.54%  bufio.(*Reader).Peek
         0     0%   100%          3  1.61%  bufio.(*Reader).Read
         0     0%   100%          2  1.08%  bufio.(*Reader).ReadSlice
         0     0%   100%          3  1.61%  bufio.(*Reader).fill
         0     0%   100%          4  2.15%  bufio.(*Writer).Flush
         0     0%   100%         12  6.45%  bytes.(*Buffer).ReadFrom

/kind bug

[rambohe-ch]

@huweihuang would you like to upload the metrics of yurthub? curl http://127.0.0.1:10267/metrics

[rambohe-ch]

@huweihuang As the flame graph, it seems that the process of writing response back to clients(like kubelet) cost most of cpu resource. and what's the overhead of machine when yurthub cost most of the cpu cores?

[huweihuang]

@huweihuang would you like to upload the metrics of yurthub? curl http://127.0.0.1:10267/metrics

# HELP node_yurthub_rejected_requests_counter counter of rejected requests for exceeding in flight limit in hub agent
# TYPE node_yurthub_rejected_requests_counter counter
node_yurthub_rejected_requests_counter 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 140899.44
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 44
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.6699264e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68906060374e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
AddKeysToAgent yes
process_virtual_memory_bytes 7.72104192e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

[huweihuang]

top info:

[rambohe-ch]

@huweihuang by the way, what's the version of the yurthub? and what's the golang version that used to compile yurthub?

[huweihuang]

@huweihuang by the way, what's the version of the yurthub? and what's the golang version that used to compile yurthub?

yurthub: projectinfo.Info{GitVersion:"-d9f74ab", GitCommit:"d9f74ab", BuildDate:"2023-07-11T02:04:11Z", GoVersion:"go1.18.10", Compiler:"gc", Platform:"linux/amd64", AllVersions:[]string{""}}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.