Add network connectivity to health-checks

[iansuvak]

Why this should be merged

Closes #447
Partially addresses #619

How this works

• Removes the TrackSubnet() call from previously called ConnectToCanonicalValidators. This was the only thing actually initializing connections and it was asynchronous and the return of that function was primarily concerned with checking the current status of the validator set and connections to them
• Fixes the bug where ConnectToCanonicalValidators was always returning as connected to all validators without checking the actual network state of completing handshakes
• Adds healthcheck function to the signatureAggregator

How this was tested

• E2E testing
• New unit test for the GetConnectedCanonicalValidators
• Manually starting up against Fuji and confirming that signature-aggregator doesn't return healthy until it's ready to aggregate signatures from the C-chain.

How is this documented

Comments

[cam-schultz]

Left a few nits, but overall this LGTM. I like the separation between network and listener health checks.

[cam-schultz]

[Not for this PR] A neat extension would be to add newly requested L1's to healthCheckSubnets, and perhaps evict them after some duration.

[iansuvak]

Thanks for bringing this up! This is a neat idea but I don't like that this would make it possible to force public facing signature-aggregator instances into unhealthy states by requesting signatures from valid subnets with unreachable nodes.

[cam-schultz]

Fair point. Adding an L1 to healthCheckSubnets only after an aggregate signature has successfully been constructed may be a fair compromise. It would be nice to support health checks for high traffic L1's that are not included in the config. Either way, we can defer to a future issue.

[bernard-avalabs]

Just have a clarification question about connected validators.

[cam-schultz]

LGTM. Just one nit