cc_cluster lifecycle test - upgrade #3018

adduarte · 2022-07-25T19:14:23Z

cc_cluster lifecycle upgrade test - upgrade

adds the required code to execute an upgrade test on a cluster

What this PR does / why we need it

Adds logic to verify class cluster upgrade for integration tests.

Which issue(s) this PR fixes

Fixes #2450

Describe testing done for PR

Release note

adds cluster upgrade integration test for tkgs

Additional information

Special notes for your reviewer

Tested the following locally:

commit 538efb762a05bc9f9e7590967cd729d168abab87 (HEAD -> cc_lifecycle_test, adduarte/cc_lifecycle_test)
Author: Adolfo Duarte <aduarte@vmware.com>
Date:   Mon Jul 25 12:03:50 2022 -0700

    cc_cluster lifecycle upgrade test - upgrade
    
    adds the required code to execute an upgrade test on a cluster

used command:

  E2E_CONFIG=/Users/aduarte/testing/tanzu-framework/trash/tkgs.yaml hack/tools/bin/ginkgo -v -trace --focus='when input file is cluster class based with CNI Antrea.*upgrade.*' pkg/v1/tkg/test/tkgctl/tkgs_cc

results:

STEP: Upgrade workload cluster "classy" in namespace "cctest"

Workload cluster 'classy' created

timeout duration of at least 15 minutes is required, using default timeout 30m0s
Creating management cluster client...
Upgrading kubernetes cluster to `v1.23.8+vmware.2-tkg.1-zshippable` version
......
worker nodes are still being upgraded for MachineDeployment 'classy-node-pool-1-h5slw', DesiredReplicas=1 Replicas=2 ReadyReplicas=2 UpdatedReplicas=1, retrying
Cluster 'classy' successfully upgraded to kubernetes version 'v1.23.8+vmware.2-tkg.1-zshippable'
Workload cluster 'classy' is being deleted 

• [SLOW TEST:1205.266 seconds]
TKGS ClusterClass based workload cluster tests
/Users/aduarte/testing/tanzu-framework/pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go:24
  when input file is cluster class based with CNI Antrea
  /Users/aduarte/testing/tanzu-framework/pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go:40
    should successfully create a cluster
    /Users/aduarte/testing/tanzu-framework/pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go:56
------------------------------
SSSS
JUnit report was created: /var/folders/3y/xfxl51_563q457kydkzw10yc0000gq/T/artifacts/junit/junit.e2e_suite.1.xml

Ran 1 of 5 Specs in 1206.075 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 4 Skipped

github-actions · 2022-07-25T19:24:22Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220725192420/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

github-actions · 2022-07-25T20:05:30Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220725200528/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-07-25T20:17:38Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220725201737/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-07-25T20:32:43Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220725203240/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

adduarte

work in progress. Functionality is there but needs refactoring where possible to avoid code clutter

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

github-actions · 2022-07-25T20:53:17Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220725205314/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

maralavi · 2022-07-27T01:16:52Z

Thanks for the PR. Changes look good, but I'm not sure if it would be feasible for automation to have two TKRs with slightly different versions of packages and check the package version is actually bumped after the upgraded.

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

haoranleo · 2022-07-27T22:04:43Z

Hi @adduarte, thanks for the change!

Currently TKGS E2E tests are not consuming E2ECommonSpec -- tests specific to TKGS are all in pkg/v1/tkg/test/tkgctl/tkgs folder. Could you move your testing code into that folder as well? Maybe add new entries in tkgs_workload_cluster_test.go for upgrade test. Thanks

maralavi · 2022-07-27T22:07:00Z

Hi @adduarte, currently TKGS E2E tests are not consuming E2ECommonSpec -- tests specific to TKGS are all in pkg/v1/tkg/test/tkgctl/tkgs folder. Could you move your testing code into that folder as well? Maybe add new entries in tkgs_workload_cluster_test.go for upgrade test. Thanks

I assume generic infrastructure-agnostic tests are still fine (and maybe even better) to be added to the e2e common.

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

haoranleo · 2022-07-27T22:10:09Z

I assume generic infrastructure-agnostic tests are still fine to be added to the e2e common.

Yeah that makes sense if the testing also works for AWS. But for TKGS test, we still need to add entry in tkgs folder for these tests to be triggered as well when we run TKGS tests. Because currently we trigger TKGS E2E tests by

hack/tools/bin/ginkgo -v -trace pkg/v1/tkg/test/tkgctl/tkgs

In the long term I agree with @maralavi that it would better if shared logic can be reused by different kind of tests.. I find it somewhat confusing too as TKGS tests are running individually.

haoranleo · 2022-07-27T22:29:36Z

Hi @adduarte, actually maybe we could continue to put the upgrade testing code in the shared package, and just add an entry in TKGS testing folder to trigger the upgrade testing.

@ggpaue's PR(#2943) does in that way. That might be much easier since the testing code might need to access to some variables or structs defined in the shared package.

adduarte · 2022-07-28T03:26:16Z

yes. that is bigger problem than just this pr :( We have been adding all our automation to e2ecommon. I think @HL-EverGreen is correct. I think is best if we add a call to E2CommonSpecInput to the tkgs_workload_cluster_test.go

github-actions · 2022-07-28T08:51:12Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220728085110/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

adduarte · 2022-07-28T09:00:24Z

Hi @adduarte, actually maybe we could continue to put the upgrade testing code in the shared package, and just add an entry in TKGS testing folder to trigger the upgrade testing.

@ggpaue's PR(#2943) does in that way. That might be much easier since the testing code might need to access to some variables or structs defined in the shared package.

Latest version of patch makes it so that everything is defined in the shared package, and the tkgs tests just make a call to it.

codecov · 2022-07-28T09:34:28Z

Codecov Report

Merging #3018 (538efb7) into main (c43d532) will decrease coverage by 0.10%.
The diff coverage is n/a.

❗ Current head 538efb7 differs from pull request most recent head 6c143ce. Consider uploading reports for the commit 6c143ce to get more accurate results

@@            Coverage Diff             @@
##             main    #3018      +/-   ##
==========================================
- Coverage   44.22%   44.11%   -0.11%     
==========================================
  Files         417      416       -1     
  Lines       42304    42225      -79     
==========================================
- Hits        18707    18629      -78     
+ Misses      21882    21878       -4     
- Partials     1715     1718       +3

Impacted Files	Coverage Δ
...ons/controllers/packageinstallstatus_controller.go	`77.99% <0.00%> (-2.32%)`	⬇️
pkg/v1/tkg/tkgpackageclient/package_update.go	`83.57% <0.00%> (-1.43%)`	⬇️
pkg/v1/tkg/client/upgrade_region.go	`24.32% <0.00%> (-1.13%)`	⬇️
addons/controllers/clusterbootstrap_controller.go	`63.28% <0.00%> (-0.62%)`	⬇️
pkg/v1/tkg/client/package_helper.go
pkg/v1/tkg/client/cluster.go	`14.43% <0.00%> (+0.73%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

adduarte · 2022-08-02T09:13:39Z

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

+			continue
+		}
+
+		specVersionIsNewer, err := isNewerVMwareVersion(tkrs[i].Spec.Version, currentTkrVersion)


if the spec version (from the tkr we are looking at) is newer (and compatible) we compare it to any version we already found, to pick the older of the two. This will ensure we pick the next avaiable version: (newer than our currentVersion, but the older tan all other newer )

github-actions · 2022-08-02T09:18:14Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220802091812/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-08-02T09:23:51Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220802092349/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go

github-actions · 2022-08-03T20:31:38Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220803203135/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-08-03T20:42:09Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220803204207/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-08-03T20:46:10Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220803204608/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-08-04T01:11:56Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220804011154/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

adduarte · 2022-08-04T06:46:48Z

Tested the following locally:

commit 346adf6da14efa2762cbc62b4a714eb1ceeaf5a1 (HEAD -> cc_lifecycle_test, adduarte/cc_lifecycle_test)
Author: Adolfo Duarte <aduarte@vmware.com>
Date:   Mon Jul 25 12:03:50 2022 -0700

    cc_cluster lifecycle upgrade test - upgrade
    
    adds the required code to execute an upgrade test on a cluster

used command:

 E2E_CONFIG=/Users/aduarte/testing/tanzu-framework/trash/tkgs.yaml hack/tools/bin/ginkgo -v -trace --focus="when input file is cluster class based with CNI Antrea" pkg/v1/tkg/test/tkgctl/tkgs_cc

results:

STEP: Upgrade workload cluster "classy" in namespace "cctest"

Workload cluster 'classy' created

timeout duration of at least 15 minutes is required, using default timeout 30m0s
Creating management cluster client...
Upgrading kubernetes cluster to `v1.23.8+vmware.2-tkg.1-zshippable` version
......
worker nodes are still being upgraded for MachineDeployment 'classy-node-pool-1-h5slw', DesiredReplicas=1 Replicas=2 ReadyReplicas=2 UpdatedReplicas=1, retrying
Cluster 'classy' successfully upgraded to kubernetes version 'v1.23.8+vmware.2-tkg.1-zshippable'
Workload cluster 'classy' is being deleted 

• [SLOW TEST:1205.266 seconds]
TKGS ClusterClass based workload cluster tests
/Users/aduarte/testing/tanzu-framework/pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go:24
  when input file is cluster class based with CNI Antrea
  /Users/aduarte/testing/tanzu-framework/pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go:40
    should successfully create a cluster
    /Users/aduarte/testing/tanzu-framework/pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go:56
------------------------------
SSSS
JUnit report was created: /var/folders/3y/xfxl51_563q457kydkzw10yc0000gq/T/artifacts/junit/junit.e2e_suite.1.xml

Ran 1 of 5 Specs in 1206.075 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 4 Skipped

github-actions · 2022-08-04T07:04:16Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220804070414/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-08-04T16:46:43Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220804164640/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go

adds the required code to execute an upgrade test on a class based cluster

github-actions · 2022-08-04T19:50:10Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220804195008/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2022-08-04T19:52:21Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/3018/20220804195218/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

adduarte requested review from a team as code owners July 25, 2022 19:14

vmwclabot added the cla-not-required label Jul 25, 2022

adduarte commented Jul 25, 2022

View reviewed changes

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

adduarte force-pushed the cc_lifecycle_test branch from f598f35 to 87c3681 Compare July 25, 2022 19:57

adduarte force-pushed the cc_lifecycle_test branch from 87c3681 to 99bb7df Compare July 25, 2022 20:07

adduarte added the do-not-merge/wip Still work in progress label Jul 25, 2022

adduarte force-pushed the cc_lifecycle_test branch from 99bb7df to 28d78f4 Compare July 25, 2022 20:23

adduarte force-pushed the cc_lifecycle_test branch from 28d78f4 to 46b1a9c Compare July 25, 2022 20:44

adduarte commented Jul 25, 2022

View reviewed changes

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

adduarte changed the title ~~WIP: cc_cluster lifecycle test - upgrade~~ cc_cluster lifecycle test - upgrade Jul 26, 2022

maralavi reviewed Jul 27, 2022

View reviewed changes

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

haoranleo reviewed Jul 27, 2022

View reviewed changes

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

adduarte force-pushed the cc_lifecycle_test branch from 46b1a9c to dcfb314 Compare July 28, 2022 08:42

adduarte force-pushed the cc_lifecycle_test branch from dcfb314 to a848e05 Compare July 28, 2022 09:39

adduarte force-pushed the cc_lifecycle_test branch from 53b9b99 to 3e0ce43 Compare August 2, 2022 09:09

adduarte commented Aug 2, 2022

View reviewed changes

adduarte force-pushed the cc_lifecycle_test branch from 3e0ce43 to 51558ad Compare August 2, 2022 09:15

maralavi reviewed Aug 3, 2022

View reviewed changes

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Show resolved Hide resolved

pkg/v1/tkg/test/tkgctl/shared/e2e_common.go Outdated Show resolved Hide resolved

adduarte force-pushed the cc_lifecycle_test branch 2 times, most recently from 700c954 to c7dc4ab Compare August 3, 2022 20:37

maralavi approved these changes Aug 3, 2022

View reviewed changes

maralavi added ok-to-merge PRs should be labelled with this before merging and removed do-not-merge/wip Still work in progress labels Aug 3, 2022

adduarte force-pushed the cc_lifecycle_test branch from c7dc4ab to 346adf6 Compare August 4, 2022 01:03

adduarte force-pushed the cc_lifecycle_test branch from 346adf6 to 538efb7 Compare August 4, 2022 06:54

adduarte force-pushed the cc_lifecycle_test branch from 538efb7 to e3f12ba Compare August 4, 2022 16:37

maralavi removed the ok-to-merge PRs should be labelled with this before merging label Aug 4, 2022

haoranleo reviewed Aug 4, 2022

View reviewed changes

pkg/v1/tkg/test/tkgctl/tkgs_cc/tkgs_cc_workload_cluster_test.go Show resolved Hide resolved

adduarte added the ok-to-merge PRs should be labelled with this before merging label Aug 4, 2022

adduarte force-pushed the cc_lifecycle_test branch from e3f12ba to 3da9c48 Compare August 4, 2022 19:41

cc_cluster lifecycle upgrade test - upgrade

6c143ce

adds the required code to execute an upgrade test on a class based cluster

adduarte force-pushed the cc_lifecycle_test branch from 3da9c48 to 6c143ce Compare August 4, 2022 19:41

adduarte merged commit 4d8a3c6 into vmware-tanzu:main Aug 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cc_cluster lifecycle test - upgrade #3018

cc_cluster lifecycle test - upgrade #3018

adduarte commented Jul 25, 2022 •

edited

Loading

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

adduarte left a comment

github-actions bot commented Jul 25, 2022

maralavi commented Jul 27, 2022 •

edited

Loading

haoranleo commented Jul 27, 2022 •

edited

Loading

maralavi commented Jul 27, 2022 •

edited

Loading

haoranleo commented Jul 27, 2022 •

edited

Loading

haoranleo commented Jul 27, 2022 •

edited

Loading

adduarte commented Jul 28, 2022

github-actions bot commented Jul 28, 2022

adduarte commented Jul 28, 2022

codecov bot commented Jul 28, 2022 •

edited

Loading

adduarte Aug 2, 2022

github-actions bot commented Aug 2, 2022

github-actions bot commented Aug 2, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 4, 2022

adduarte commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

cc_cluster lifecycle test - upgrade #3018

cc_cluster lifecycle test - upgrade #3018

Conversation

adduarte commented Jul 25, 2022 • edited Loading

What this PR does / why we need it

Which issue(s) this PR fixes

Describe testing done for PR

Release note

Additional information

Special notes for your reviewer

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

github-actions bot commented Jul 25, 2022

adduarte left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 25, 2022

maralavi commented Jul 27, 2022 • edited Loading

haoranleo commented Jul 27, 2022 • edited Loading

maralavi commented Jul 27, 2022 • edited Loading

haoranleo commented Jul 27, 2022 • edited Loading

haoranleo commented Jul 27, 2022 • edited Loading

adduarte commented Jul 28, 2022

github-actions bot commented Jul 28, 2022

adduarte commented Jul 28, 2022

codecov bot commented Jul 28, 2022 • edited Loading

Codecov Report

adduarte Aug 2, 2022

Choose a reason for hiding this comment

github-actions bot commented Aug 2, 2022

github-actions bot commented Aug 2, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 4, 2022

adduarte commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

adduarte commented Jul 25, 2022 •

edited

Loading

maralavi commented Jul 27, 2022 •

edited

Loading

haoranleo commented Jul 27, 2022 •

edited

Loading

maralavi commented Jul 27, 2022 •

edited

Loading

haoranleo commented Jul 27, 2022 •

edited

Loading

haoranleo commented Jul 27, 2022 •

edited

Loading

codecov bot commented Jul 28, 2022 •

edited

Loading