rework configuration using custom resources

[klihub]

This patch set reworks how configuration is handled, switching from a single dynamically constructed map[string]interface{}/ConfigMap to multiple/per-plugin statically defined custom resources. In particular the patch set,

• defines a v1alpha1 configuration API (per (resource policy) plugin custom resource for configuration)
• implements a new agent for monitoring configuration, delivering configuration data
• updates resource manager to wait for initial configuration before starting up
• updates resource manager to reconfigure the various components for subsequent configuration changes
• updates the policy plugins to use the new CR-based configuration
• reworks the helm charts to deal with custom resources instead of ConfigMaps
• reworks the e2e test infra to use helm/charts instead of deployment files
• updates the test cases for the new configuration mechanism and format(s)

Note: Although already due to its mere size this is still work in progress, I've given up the idea that we'd be able to get this in final shape in a single go.

TODO:

• externally visible changes could use a consistency check by reviewers (config data type, names, etc. also wrt. to other existing user-visible interfaces, like namespace used for annotations, etc.)
• large parts of the code could use a cleanup (but maybe other than the worst could be addressed with subsequent PRs?)
• nuking a bunch of old code probably would now allow further structural and other simplifications (maybe in subsequent PRs?)

[klihub]

One way to give this a test using the Helm charts is to generate the images, copy and import them to the test node (podman or ctr depending on your runtime), then use a Helm override like this with the name/tag matching your build and how you imported the image...

# Test overrides for TA Helm Chart. You can use an almost identical one for balloons.
#   - assume/use pre-imported plugin image
#   - never try to pull
#   - turn on full debugging
---
image:
  name: localhost/nri-resource-policy-topology-aware
  tag: v0.1.0-138-g0c402fe
  pullPolicy: Never

policyConfig:
  reservedResources:
    cpu: 750m

commonConfig:
  control:
    enable:
      - cpu
  log:
    debug:
      "*": true
    logSource: true
    klog:
      skip_headers: true
  instrumentation:
    reportPeriod: 60s
    samplingRatePerMillion: 0

hostPort: 8891

resources:
  cpu: 500m
  memory: 512Mi

nri:
  patchRuntimeConfig: false

initContainerImage:
  name: ghcr.io/containers/nri-plugins/nri-config-manager
  # If not defined Chart.AppVersion will be used
  #tag: unstable
  pullPolicy: Always

tolerations: []

[marquiz]

Thank you @klihub for this huge effort 👏 I think the CRD API now looks nice and clean. And everything else (if we find something) we can easily change later without touching the UI (I hope) 😇

LGTM

[askervin]

Thanks for this huge work, @klihub!

LGTM.