Error from server (could not resolve TKR/OSImage for machineDeployments: [tkg-md-1 tkg-md-2]

[ydp]

Bug description

When create CC cluster on vSphere with prodcc plan, it reports error

Error: unable to set up management cluster: unable to create management cluster: unable to apply cluster configuration: kubectl apply failed, output: secret/mgmt-vsphere-credential configured
secret/mgmt-nsxt-credential configured
vspherecpiconfig.cpi.tanzu.vmware.com/mgmt unchanged
clusterbootstrap.run.tanzu.vmware.com/mgmt configured
secret/mgmt configured
Error from server (could not resolve TKR/OSImage for machineDeployments: [tkg-md-1 tkg-md-2], query: {controlPlane: {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'}, machineDeployments: [{k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: ''} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: ''}]}, result: {controlPlane: {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]}, machineDeployments: [{k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.1-3e6b7a84930e5368c38aa867f998ced0 v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7 v1.23.8---vmware.2-tkg.1-85a434f93857371fccb566a414462981]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.1-3e6b7a84930e5368c38aa867f998ced0 v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7 v1.23.8---vmware.2-tkg.1-85a434f93857371fccb566a414462981]]}]}): error when creating "/tmp/kubeapply-1641870247": admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: could not resolve TKR/OSImage for machineDeployments: [tkg-md-1 tkg-md-2], query: {controlPlane: {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'}, machineDeployments: [{k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=photon'} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: ''} {k8sVersionPrefix: 'v1.23.8+vmware.2-tkg.1-zshippable', tkrSelector: '', osImageSelector: ''}]}, result: {controlPlane: {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]}, machineDeployments: [{k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.1-3e6b7a84930e5368c38aa867f998ced0 v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7 v1.23.8---vmware.2-tkg.1-85a434f93857371fccb566a414462981]]} {k8sVersion: 'v1.23.8+vmware.2', tkrName: 'v1.23.8---vmware.2-tkg.1-zshippable', osImagesByTKR: map[v1.23.8---vmware.2-tkg.1-zshippable:[v1.23.8---vmware.2-tkg.1-3e6b7a84930e5368c38aa867f998ced0 v1.23.8---vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7 v1.23.8---vmware.2-tkg.1-85a434f93857371fccb566a414462981]]}]}

Affected product area (please put an X in all that apply)

• ( ) APIs
• ( ) Addons
• (*) CLI
• ( ) Docs
• ( ) IAM
• ( ) Installation
• ( ) Plugin
• ( ) Security
• ( ) Test and Release
• ( ) User Experience
• ( ) Developer Experience

Expected behavior

Create CC cluster with prodcc plan on vSphere succeed.

Steps to reproduce the bug

Use prodcc plan to create vSphere CC cluster

Version (include the SHA if the version is not obvious)

Environment where the bug was observed (cloud, OS, etc)

Relevant Debug Output (Logs, manifests, etc)