Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

发现Poalris Server运行状态正常情况下,仍存在客户端服务实例注册异常,同时检查客户端服务实例运行正常 #1395

Open
flashbai opened this issue Nov 15, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@flashbai
Copy link

Describe the bug

  1. 问题现象:在polaris server组件内存未溢出、CPU未打满、polaris-server服务pod运行正常未发生重启情况下,发现客户端服务实例注册polaris异常,同时在客户端服务的相关日志中发现如下报错:
    image
  • 另外Polaris Server发现内存一直上涨,但一直没有超过mem limit的8G限制
  • 同时将Polaris server服务Pod的replica副本数调整为1后,再重启polaris server,发现客户端服务注册正常,同时观察后续Polaris server内存也没有上涨;
  • 另外发现3副本时Polaris server中存在如下报错信息:
    polaris-health-check-error

To Reproduce
Steps to reproduce the behavior.

  1. Polaris Server部署版本:1.18.1
  2. 部署方式:Polaris Server组件通过K8S Helm部署,3副本,cpu limit为4c, mem limit为8g;
  3. Polaris监控:检查Polaris Server组件cpu、mem、网卡流量、负载等相关监控指标正常,唯一异常点是内存监控指标持续上涨,但内存使用率未到limit上限,同时没有发生过OOM;
  4. 客户端服务与polaris部署在同一个k8s集群,同时客户端服务配置1s超时
  5. 检查客户端服务pod与polaris server pod之间的网络延迟不高,同时没有网络抖动;
    Expected behavior
    A clear and concise description of what you expected to happen.

Environment

  • Version: polaris-server版本1.18.1
  • OS: 宿主机系统Ubuntu18
    Additional context
    Add any other context about the problem here.
@flashbai flashbai added the bug Something isn't working label Nov 15, 2024
@flashbai
Copy link
Author

目前的话Polaris-Server在单副本情况下运行正常,调整为多副本后,运行一段时间,就会出现客户端服务注册异常情况

@flashbai
Copy link
Author

polaris-server pod在内存limit上限为8G,内存持续上涨的问题在调整为pod为单副本并重启pod后恢复正常,但调整为多副本就会出现内存一直上涨问题
Uploading polaris-server pod内存使用率持续上涨在调整为单副本并重启后恢复正常.png…

@flashbai
Copy link
Author

客户端注册polaris时的1s超时导致注册异常截图
Uploading image.png…

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant