Does CNTI support Helm charts with nested dependencies?

[adityaSiso]

Scenario
Our Helm chart setup involves multiple nested levels (3-4 layers deep). While attempting to generate a cnf-testsuite.yml for the chart, we encountered the error mentioned below.

There are not adequate logs when using the -l DEBUG option. And when deploying the same chart via helm command it seems to be working fine.

Is there any limitation with the tool for recurring level charts or there is something else?

>>> cnf-testsuite generate_config config-src=cnf_chart output-file=./cnf-testsuite.yml
E, [2024-11-19 11:36:48 +00:00 #542269] ERROR -- cnf-testsuite: ./cnf-testsuite.yml already exists
/usr/share/crystal/src/yaml/pull_parser.cr:339:7 in 'raise'
/usr/share/crystal/src/yaml/pull_parser.cr:122:7 in 'read_next'
/usr/share/crystal/src/gc/boehm.cr:129:5 in 'parse'
/usr/share/crystal/src/yaml/schema/core.cr:8:24 in 'parse'
/workspace/lib/helm/helm.cr:80:9 in 'parse_manifest_as_ymls'
/workspace/lib/helm/helm.cr:74:5 in 'export_manifest'
/workspace/src/tasks/utils/generate_config.cr:13:5 in 'generate_config'
/workspace/lib/sam/src/sam/execution.cr:20:7 in '__crystal_main'
/usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
src/env/__libc_start_main.c:95:2 in 'libc_start_main_stage2'
did not find expected alphabetic or numeric character at line 1, column 2, while scanning an alias at line 1, column 1

Tools Info:
cnti version: v.1.1.0 (Required airgap support)
helm version: v3.15.2

[svteb]

The error seems to be caused by the fact that you had a cnf-testsuite.yml file already present in the . directory. Could you rename the output file/delete the other cnf-testsuite.yml and attempt to run it again? It is hard to determine if the exception is caused by the error or not. Also did you attempt to manually construct the config?

[adityaSiso]

The error seems to be caused by the fact that you had a cnf-testsuite.yml file already present in the . directory. Could you rename the output file/delete the other cnf-testsuite.yml and attempt to run it again?

We are getting the same issue after deleting the already present cnf-testsuite.yml file.

>>> cnf-testsuite generate_config config-src=eric-ccdm output-file=./cnf-testsuite.yml
/usr/share/crystal/src/yaml/pull_parser.cr:339:7 in 'raise'
/usr/share/crystal/src/yaml/pull_parser.cr:122:7 in 'read_next'
/usr/share/crystal/src/gc/boehm.cr:129:5 in 'parse'
/usr/share/crystal/src/yaml/schema/core.cr:8:24 in 'parse'
/workspace/lib/helm/helm.cr:80:9 in 'parse_manifest_as_ymls'
/workspace/lib/helm/helm.cr:74:5 in 'export_manifest'
/workspace/src/tasks/utils/generate_config.cr:13:5 in 'generate_config'
/workspace/lib/sam/src/sam/execution.cr:20:7 in '__crystal_main'
/usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
src/env/__libc_start_main.c:95:2 in 'libc_start_main_stage2'
did not find expected alphabetic or numeric character at line 1, column 2, while scanning an alias at line 1, column 1

Also did you attempt to manually construct the config?

Yes, manually created the config file and tried setting up the cnf but it fails with the same below logs:

>>> cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
Successfully created directories for cnf-testsuite
Created cnf-testsuite namespace on the Kubernetes cluster
cnf setup online mode
/usr/share/crystal/src/yaml/pull_parser.cr:339:7 in 'raise'
/usr/share/crystal/src/yaml/pull_parser.cr:122:7 in 'read_next'
/usr/share/crystal/src/gc/boehm.cr:129:5 in 'parse'
/usr/share/crystal/src/yaml/schema/core.cr:8:24 in 'parse'
/workspace/lib/helm/helm.cr:80:9 in 'parse_manifest_as_ymls'
/workspace/src/tasks/utils/cnf_manager.cr:952:15 in 'sample_setup'
/usr/share/crystal/src/colorize.cr:132:3 in '->'
/workspace/lib/sam/src/sam/execution.cr:20:7 in '__crystal_main'
/usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
src/env/__libc_start_main.c:95:2 in 'libc_start_main_stage2'
did not find expected alphabetic or numeric character at line 1, column 2, while scanning an alias at line 1, column 1

config file:

>>> cat cnf-testsuite.yml
---
release_name: eric-ccdm-1732793278
helm_directory: eric-ccdm
helm_install_namespace: cnfspace-testing

[adityaSiso]

The CNF we are trying to test has 3-4 levels of dependent charts in the directory.

[svteb]

If I understand it correctly the -l debug option did not provide any additional logging. Could you try to run export LOG_LEVEL=debug or export LOGLEVEL=debug (if these fail switch out debug for info, hard to say what is and is not working in this version) and run cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml again. If the output changes in any way could you please provide it (you can send it to my email slavo.valko@tietoevry.com in case there might be some sensitive information). If the output does not change I'll attempt further digging, these old versions of testsuite are quite the headache.

As a side note: The helm library went through quite the rewrite in recent updates and on the newest version I managed to deploy a nested helm chart (Kafka with the zookeeper dependency). I believe that there was not that big of a change in terms of the logic so nested helm charts should probably work even on the older versions.

[adityaSiso]

From looking at the logs, I can ensure that -l debug and export LOG_LEVEL=debug works the same way. There is no change in the output.

[adityaSiso]

It would be very helpful if you could specify the CNTI version that fully supports airgap testing.

[kosstennbl]

Hi, airgap support was fully removed in CNF-Testsuite 1.3.2
Also, config generation methods are now changed, see 1.4.0-beta release.
If your CNF requires multiple separate charts to be installed, there is new possibility in configuration for that in 1.4.0
And if Helm chart has dependencies - that shouldn't be a problem if the main Chart.yaml is able to be installed via one "helm install" command

[svteb]

It seems that I cannot replicate the issue. I've managed to deploy a helm chart with nested dependencies on the version v1.1.0. I also don't quite see the reason as to why you do not get any logs, there should be at least some info logs right at the beginning before the exception is raised:

export LOG_LEVEL=info

./cnf-testsuite version
CNF TestSuite version: v1.1.0
I, [2024-11-29 09:01:20 +00:00 #2456993]  INFO -- cnf-testsuite: VERSION: v1.1.0

./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
Successfully created directories for cnf-testsuite
I, [2024-11-29 09:01:02 +00:00 #2456788]  INFO -- cnf-testsuite: Globally installed helm satisfies required version. Skipping local helm install.
I, [2024-11-29 09:01:02 +00:00 #2456788]  INFO -- cnf-testsuite: KubectlClient::Create.namespace command: kubectl create namespace cnf-testsuite
cnf-testsuite namespace already exists on the Kubernetes cluster
...

Do you get the info log output upon running ./cnf-testsuite version ? If so, the issue might be with the exception blocking the log flushing, could you attempt to run stdbuf -oL ./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml (this should hopefully force the logs to be flushed).

[adityaSiso]

Do you get the info log output upon running ./cnf-testsuite version?

Yes we were getting info logs.

If so, the issue might be with the exception blocking the log flushing, could you attempt to run stdbuf -oL ./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml (this should hopefully force the logs to be flushed).

Tried it encountering the same issue.

[svteb]

Alright, here’s where we stand. Since airgap mode is no longer supported in the latest versions, and we’re not maintaining older releases, this might need to be left unresolved. I highly recommend finding a way to use the latest version, but I understand that might not be possible in your case. Alternatively, you’re welcome to submit a pull request to reintroduce airgap mode in latest version if it’s important for your setup.

If you’re sticking with v1 I cannot help you without the logs, you’ll probably have to clone the repo, tweak the logging code, and build your own binary. Here are the changes you can make to ensure logs are flushed properly (if this doesn’t work, I’m honestly out of ideas):

In src/tasks/utils/utils.cr:

1. Add this at the top of the file, right after the require statements:

STDOUT.sync = true
STDERR.sync = true
at_exit { Log.flush }

2. Update the Log.setup calls from this:

Log.setup(Log::Severity::Error, Log::IOBackend.new(formatter: log_formatter))
Log.setup(loglevel, Log::IOBackend.new(formatter: log_formatter))

to this:

Log.setup(Log::Severity::Error, Log::IOBackend.new(formatter: log_formatter, sync: true))
Log.setup(loglevel, Log::IOBackend.new(formatter: log_formatter, sync: true))

That should help ensure logs are printed out. However, even if we fix the logging issue, if the underlying bug with nested helm chart dependencies is within the testsuite itself, the chances of us releasing a fix for v1 are very slim. I’ll still try to help diagnose the issue, but resolving it will likely require some effort on your end.

[adityaSiso]

Okay, We will see if we can migrate to the latest version or maybe find a way to resolve the issue.

Thanks for the support @svteb.