Daemon commands become increasingly slow over time

[brechtcs]

OS: NixOS 20.03, Linux kernel 5.4.41

Node version: 13.8.0, installed from NPM

I'm noticing that if the daemon has been running for a while, the CLI becomes increasingly slow. A simple hyperdrive status can take over a minute to complete. Stopping and restarting the daemon solves the issue, so I'm thinking it might be a memory issue.

I haven't been able to consistently reproduce it yet, but I'm under the impression that frequent use of the export command might have something to do with it. (I haven't been able to configure FUSE yet.)

Another option is that it's caused by frequent connections from a remote machine, which is also something I've been doing frequently when testing hyperup.

$ hyperdrive status
FUSE is not available on your platform.
The Hyperdrive daemon is running:

  API Version:             0
  Daemon Version:          1.13.12
  Client Version:          1.15.1
  Schema Version:          1.10.0
  Hyperdrive Version:      10.11.0
  Fuse Native Version:
  Hyperdrive Fuse Version:

  Holepunchable:           true
  Remote Address:          <redacted>:49737

  Fuse Available:          false
  Fuse Configured:         false

  Uptime:                  0 Days 5 Hours 11 Minutes 37 Seconds

[andrewosh]

Hey @brechtcs thanks for reporting this. In 1.13.13 we added periodic mem/cpu stats logging, which should help us trace any potential leaks. We've seen a few other issues related to memory usage so connecting it to the export command (potentially) could be a good tip. As could the frequent reconnections -- most of our use-cases involve long-running services, like Beaker, maintaining a sinlge connection.

The log file in 1.13.13 has also been scrubbed of any sensitive keys, so it's safe to upload -- then we can look at the stats over time.

Thanks again for the report.

[brechtcs]

Cool, I'll update the daemon and report back.

[brechtcs]

Quick update, I haven't really noticed this anymore since #69 started occuring, so maybe they're related? I guess it's equally possible that I'm not reaching the necessary uptime for this to become an issue though, so I'll keep monitoring it.