Feat: add kapp on kfd.yaml, update apply scripts

[nutellinoit]

Summary 💡

This PR introduces the application of manifests generated by kustomize (in both distribution and plugins phases) using the kapp CLI. Kapp allows for applying manifests and verifying that everything being installed is functioning correctly. It can also apply CRDs, wait for them to be available, and then apply the referring CRs. This significantly reduces and simplifies the complexity of apply operations, which were previously performed with plain kubectl.

Here's a breakdown of the changes:

• kustomize manifests application: Instead of using kubectl apply, we now leverage kapp to manage manifest deployments.
• Verification and functionality: Kapp ensures that all deployed resources are functioning correctly.
• CRD handling: Kapp simplifies the process of applying CRDs and their associated CRs.
• Future migration benefits: By utilizing kapp's state-based resource removal capabilities, migrations will be streamlined.

This PR brings significant improvements to our deployment and migration workflows by leveraging the powerful features of kapp.

Tests

The following manual tests were performed:

• Tested cluster upgrade from KFD 1.30.0 (OnPremises provider) to KFD 1.31.0 with:
  • Loki
  • Opensearch
  • Kyverno
  • Gatekeeper
  • Tempo

Future work

This feature will allow for reduced migrations complexity in the future as kapp is capable of removing resources from the cluster leveraging a state (a secret), similar to helm.