redo of BUILDKIT caching PR

[jayunit100]

What this PR does / why we need it

fixes #2730

This is a "redo" of https://github.com/vmware-tanzu/tanzu-framework/pull/2731/files where @randomvariable originally did some caching.
Only differences, haven't tested...

• Tried to make it all one variable $CACHE
• feature test controllers moved , updated
• pinniped split off to a new place, updated that
• v2 directory from Ivans mr is now split out to other dirs , so I updated those parts

Release Note

Enable BUILDKIT caching for Tanzu Framework build

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20221122135528/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20221122152352/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[ameukam]

Suggested change

	ARG CACHE_GOLANG=cache,target=/root/.local/share/golang
	ARG CACHE_GOLANG=type=cache,target=/root/.local/share/golang

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20221122212228/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[jayunit100]

=> [internal] load build definition from Dockerfile                                                                                                                                                                                                                             0.0s
 => => transferring dockerfile: 1.57kB                                                                                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                0.0s
 => => transferring context: 34B                                                                                                                                                                                                                                                 0.0s
 => [internal] load metadata for gcr.io/distroless/static:nonroot                                                                                                                                                                                                                0.5s
 => [internal] load metadata for docker.io/library/golang:1.18                                                                                                                                                                                                                   1.6s
 => [auth] library/golang:pull token for registry-1.docker.io                                                                                                                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                                                                                9.4s
 => => transferring context: 526.93MB                                                                                                                                                                                                                                            9.3s
 => [builder 1/5] FROM docker.io/library/golang:1.18@sha256:0936e7443a0466cdad467ab8bafe6082999c065ebc389e7db22b0452d5d98842                                                                                                                                                     0.0s
 => CACHED [stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:ed05c7a5d67d6beebeba19c6b9082a5513d5f9c3e22a883b9dc73ec39ba41c04                                                                                                                                           0.0s
 => CACHED [builder 2/5] WORKDIR /workspace                                                                                                                                                                                                                                      0.0s
 => [builder 3/5] COPY ./ ./                                                                                                                                                                                                                                                     3.7s
 => [builder 4/5] RUN --mount=type=cache,target=/root/.cache/go-build --mount=type=cache,target=/root/.local/share/golang --mount=type=cache,target=/go/pkg/mod go mod download                                                                                                177.0s
 => [builder 5/5] RUN --mount=type=cache,target=/root/.cache/go-build --mount=type=cache,target=/root/.local/share/golang --mount=type=cache,target=/go/pkg/mod cd addons && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -ldflags "-s -w -X 'github.com/vmwa  62.1s
 => [stage-1 2/3] COPY --from=builder /workspace/addons/bin/manager .                                                                                                                                                                                                            0.1s 
 => exporting to image                                                                                                                                                                                                                                                           0.2s 
 => => exporting layers                                                                                                                                                                                                                                                          0.2s 
 => => writing image sha256:117189312447435f6815b92612ac0545db9abb9cbf6f5015e0300374b514b894                                                                                                                                                                                     0.0s 
 => => naming to projects.registry.vmware.com/tanzu_framework/addons-controller:v0.28.0-dev-151-g150131b7                                                                                                                                                                        0.0s 
                                                                                                                                                                                                                                                                                      
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
/Library/Developer/CommandLineTools/usr/bin/make -C cliplugins docker-build
cd /Users/jayunit100/SOURCE/tanzu-framework/artifacts/linux/amd64/cli/#/v0.28.0-dev && docker build -t projects.registry.vmware.com/tanzu_framework/tanzu-cli-plugins/#-linux-amd64:latest -f /Users/jayunit100/SOURCE/tanzu-framework/cliplugins/Dockerfile.templates .
/bin/sh: line 0: cd: /Users/jayunit100/SOURCE/tanzu-framework/artifacts/linux/amd64/cli/#/v0.28.0-dev:

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20221123075409/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20221123165006/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[codecov]

Codecov Report

Merging #3972 (4f83805) into main (af08710) will decrease coverage by 0.79%.
The diff coverage is 90.21%.

@@            Coverage Diff             @@
##             main    #3972      +/-   ##
==========================================
- Coverage   49.51%   48.72%   -0.79%     
==========================================
  Files         452      482      +30     
  Lines       44986    47164    +2178     
==========================================
+ Hits        22273    22979     +706     
- Misses      20588    22013    +1425     
- Partials     2125     2172      +47     

Impacted Files	Coverage Δ
tkg/client/validate.go	61.50% <0.00%> (-0.12%)	⬇️
tkg/client/cluster.go	35.34% <88.88%> (+2.05%)	⬆️
tkg/client/upgrade_cluster.go	46.75% <91.66%> (+0.90%)	⬆️
tkg/managementcomponents/helper.go	92.61% <97.05%> (+0.68%)	⬆️
tkg/client/kubevip_update_helper.go	77.85% <100.00%> (+3.91%)	⬆️
tkg/clusterclient/clusterclient.go	49.14% <100.00%> (ø)
...ons/controllers/packageinstallstatus_controller.go	79.15% <0.00%> (ø)
...isolated-cluster/imagepullop/publishimagestotar.go	61.36% <0.00%> (ø)
cmd/cli/plugin/cluster/get_machinehealthcheck.go	11.42% <0.00%> (ø)
... and 33 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20221123203109/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[rajaskakodkar]

rebasing this PR

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20230216175308/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3972/20230217221655/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[sriramandev]

lgtm