PMM 2.44.0 Mongo DB too many connections

[GenLN]

Description

Hi we noticed after upgrade that pmm client keeps opening new connections on mongodb nodes. As we had like around 8-9k connections per server by pmm client.

Expected Results

Not too many connections by pmm client to mongodb

Actual Results

Too many connections from pmm client and it keep opening a new one.

Version

PMM Server 2.44.0, PMM agent 2.44.0

Steps to reproduce

Run mongo cluster with many collections. Connect pmm client and pmm server, same version 2.44.0

Relevant logs

Code of Conduct

• I agree to follow Percona Community Code of Conduct

[BupycHuk]

Hi @GenLN can you provide logs? This problem existed in PMM 2.43.0 and 2.43.1, are you sure that you updated PMM Client?

[GenLN]

time="2025-01-22T10:49:37.377+00:00" level=warning msg="time=\"2025-01-22T10:49:37Z\" level=warning msg=\"failed to create PBM client: create mongo connection: ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: server1:27017, Type: RSSecondary, Average RTT: 2526081 }, { Addr: server2:27017, Type: Unknown, Last error: dial tcp <IP>:27017: connect: connection refused }, { Addr: server3:27017, Type: RSSecondary, Average RTT: 2370255 }, ] }\" collector=pbm" agentID=/agent_id/<agentID> component=agent-process type=mongodb_exporter

$ pmm-agent --version
ProjectName: pmm-agent
Version: 2.44.0
PMMVersion: 2.44.0

docker env

These logs keep repeating on primary nodes of mongodb, and connections grows on secondary nodes...
After we remove pmm client from primary mongodb node, all connections closed.

[iivank0vic]

Hello @BupycHuk

We have the same problem on PMM 3.0.0 (server and client)

Connections grow on secondary nodes, after we restart the PMM client from the primary MongoDB node, all connections are closed.

pmm-agent --version
ProjectName: pmm-agent
Version: 3.0.0
PMMVersion: 3.0.0
Timestamp: 2025-01-23 13:50:41 (UTC)
FullCommit: d989d76c744b70da0a0ff9fbdc58010d3bc8c344

(Before and after restarting the PMM agent on the primary node)

PMM Version 2.41

• Not affected
• Don't have pbm-agent check

pbm-logs on primary

[BupycHuk]

Hi, can you provide information about your topology and if you have PBM installed and configured?

[iivank0vic]

Hello @BupycHuk

We do not use PBM at all. We are using the in-house PXC docker image and PMMclient docker image.
For backups, we rely on our custom scripts. Feel free to let me know if you need any specific details.