Integration tests failing for kfp bundle track/2.0-alpha.7

[misohu]

Bug Description

Integration tests failing for kfp bundle track/2.0-alpha.7

To Reproduce

1. clone repo
2. git checkout track/2.0-alpha.7
3. tox -e bundle-integration -- --model kubeflow --destructive-mode

Environment

juju - 2.9.47-ubuntu-amd64
microk8s - v1.24.17

Relevant Log Output

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.2, pluggy-1.3.0 -- /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/bin/python
cachedir: .tox/bundle-integration/.pytest_cache
rootdir: /home/runner/work/kfp-operators/kfp-operators
configfile: pyproject.toml
plugins: operator-0.29.0, asyncio-0.21.1
asyncio: mode=strict
collecting ... collected 1 item

tests/integration/test_bundle.py::test_build_and_deploy FAILED

=================================== FAILURES ===================================
____________________________ test_build_and_deploy _____________________________
Traceback (most recent call last):
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/kfp-operators/kfp-operators/tests/integration/test_bundle.py", line 63, in test_build_and_deploy
    await ops_test.model.wait_for_idle(
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/juju/model.py", line 2613, in wait_for_idle
    _raise_for_status(errors, "error")
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/juju/model.py", line 2525, in _raise_for_status
    raise error_type("{}{} in {}: {}".format(
juju.errors.JujuUnitError: Unit in error: istio-pilot/0
------------------------------ Captured log setup ------------------------------
INFO     pytest_operator.plugin:plugin.py:675 Connecting to existing model github-pr-3b62d-microk8s:kubeflow on unspecified cloud
------------------------------ Captured log call -------------------------------
INFO     test_bundle:test_bundle.py:38 Building charms for: {'kfp-api': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-api'), 'kfp-persistence': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-persistence'), 'kfp-profile-controller': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-profile-controller'), 'kfp-schedwf': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-schedwf'), 'kfp-ui': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-ui'), 'kfp-viewer': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-viewer'), 'kfp-viz': PosixPath('/home/runner/work/kfp-operators/kfp-operators/charms/kfp-viz')}
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-api
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-persistence
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-profile-controller
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-schedwf
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-ui
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-viewer
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-viz
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-viewer in 99.82s
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-schedwf in 103.30s
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-profile-controller in 161.33s
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-persistence in 162.16s
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-viz in 170.05s
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-ui in 171.25s
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-api in 214.96s
INFO     test_bundle:test_bundle.py:40 Built charms: {'kfp-api': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-api_ubuntu-20.04-amd64.charm'), 'kfp-persistence': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-persistence_ubuntu-20.04-amd64.charm'), 'kfp-profile-controller': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-profile-controller_ubuntu-20.04-amd64.charm'), 'kfp-schedwf': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-schedwf_ubuntu-20.04-amd64.charm'), 'kfp-ui': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-ui_ubuntu-20.04-amd64.charm'), 'kfp-viewer': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-viewer_ubuntu-20.04-amd64.charm'), 'kfp-viz': PosixPath('/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/charms/kfp-viz_ubuntu-20.04-amd64.charm')}
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0
INFO     test_bundle:test_bundle.py:55 Deploying bundle to github-pr-3b62d-microk8s:kubeflow using cmd 'juju deploy -m github-pr-3b62d-microk8s:kubeflow --trust /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/tmp/pytest/kubeflow0/bundles/bundle.yaml'
INFO     juju.model:model.py:2621 Waiting for model:
  argo-controller/0 [executing] blocked: Waiting for object-storage relation data
  istio-ingressgateway/0 [executing] blocked: Please add required relation to istio-pilot
  istio-pilot/0 [executing] maintenance: Deploying Istio control plane with Istio CNI plugin.
  kfp-api/0 [allocating] waiting: agent initializing
  kfp-db/0 [allocating] waiting: installing agent
  kfp-persistence/0 [allocating] waiting: installing agent
  kfp-profile-controller/0 [allocating] waiting: installing agent
  kfp-schedwf/0 [allocating] waiting: installing agent
  kfp-ui/0 [allocating] waiting: installing agent
  kfp-viewer/0 [allocating] waiting: installing agent
  kfp-viz/0 [allocating] waiting: installing agent
  kubeflow-dashboard/0 [allocating] waiting: installing agent
  kubeflow-profiles/0 [allocating] waiting: installing agent
  metacontroller-operator/0 [allocating] waiting: installing agent
  minio/0 [allocating] waiting: installing agent
---------------------------- Captured log teardown -----------------------------
INFO     pytest_operator.plugin:plugin.py:783 Model status:

Model     Controller                Cloud/Region        Version  SLA          Timestamp
kubeflow  github-pr-3b62d-microk8s  microk8s/localhost  2.9.47   unsupported  08:03:27Z

App                      Version              Status   Scale  Charm                    Channel        Rev  Address         Exposed  Message
argo-controller                               blocked      1  argo-controller          3.3/stable     376                  no       Waiting for object-storage relation data
istio-ingressgateway                          waiting      1  istio-gateway            latest/edge    837  10.152.183.240  no       List of <ops.model.Relation istio-pilot:11> versions not found for apps: istio-pilot
istio-pilot                                   error        1  istio-pilot              latest/edge    813  10.152.183.6    no       hook failed: "istio-pilot-relation-created"
kfp-api                                       waiting      1  kfp-api                                   0  10.152.183.75   no       installing agent
kfp-db                   mariadb/server:10.3  waiting      1  charmed-osm-mariadb-k8s  latest/stable   35  10.152.183.31   no       waiting for container
kfp-persistence                               waiting    0/1  kfp-persistence                           0                  no       agent initializing
kfp-profile-controller                        waiting    0/1  kfp-profile-controller                    0                  no       installing agent
kfp-schedwf                                   waiting    0/1  kfp-schedwf                               0                  no       installing agent
kfp-ui                                        waiting    0/1  kfp-ui                                    0                  no       installing agent
kfp-viewer                                    waiting    0/1  kfp-viewer                                0                  no       installing agent
kfp-viz                                       waiting    0/1  kfp-viz                                   0                  no       installing agent
kubeflow-dashboard                            waiting    0/1  kubeflow-dashboard       latest/edge    479                  no       installing agent
kubeflow-profiles                             waiting    0/1  kubeflow-profiles        latest/edge    362                  no       installing agent
metacontroller-operator                       waiting    0/1  metacontroller-operator  latest/edge    250                  no       installing agent
minio                                         waiting    0/1  minio                    latest/edge    292                  no       installing agent

Unit                       Workload  Agent       Address       Ports     Message
argo-controller/0*         blocked   idle                                Waiting for object-storage relation data
istio-ingressgateway/0*    waiting   idle        10.1.166.203            List of <ops.model.Relation istio-pilot:11> versions not found for apps: istio-pilot
istio-pilot/0*             error     idle        10.1.166.204            hook failed: "istio-pilot-relation-created"
kfp-api/0*                 waiting   executing   10.1.166.207            (config-changed) List of <ops.model.Relation object-storage:5> versions not found for apps: minio
kfp-db/0*                  waiting   idle                      3306/TCP  waiting for container
kfp-persistence/0*         waiting   allocating                          agent initializing
kfp-profile-controller/0   waiting   allocating                          installing agent
kfp-schedwf/0              waiting   allocating                          installing agent
kfp-ui/0                   waiting   allocating                          installing agent
kfp-viewer/0               waiting   allocating                          installing agent
kfp-viz/0                  waiting   allocating                          installing agent
kubeflow-dashboard/0       waiting   allocating                          installing agent
kubeflow-profiles/0        waiting   allocating                          installing agent
metacontroller-operator/0  waiting   allocating                          installing agent
minio/0                    waiting   allocating                          installing agent


INFO     pytest_operator.plugin:plugin.py:789 Juju error logs:

unit-istio-pilot-0: 08:03:23 ERROR unit.istio-pilot/0.juju-log istio-pilot:11: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1224, in <module>
    main(Operator)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 367, in reconcile
    self._reconcile_gateway()
  File "./src/charm.py", line 681, in _reconcile_gateway
    "port": self._gateway_port,
  File "./src/charm.py", line 521, in _gateway_port
    if self._use_https():
  File "./src/charm.py", line 930, in _use_https
    return self._use_https_with_tls_provider() or self._use_https_with_tls_secret()
  File "./src/charm.py", line 942, in _use_https_with_tls_secret
    secret = self.model.get_secret(label=TLS_SECRET_LABEL)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/model.py", line 272, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/model.py", line 3148, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/model.py", line 2785, in _run
    raise RuntimeError(f'command not found: {args[0]}')
RuntimeError: command not found: secret-get
unit-istio-pilot-0: 08:03:23 ERROR juju.worker.uniter.operation hook "istio-pilot-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-kfp-api-0: 08:03:24 ERROR unit.kfp-api/0.juju-log Failed to handle <LeaderElectedEvent via KfpApiOperator/on/leader_elected[31]> with error: List of <ops.model.Relation object-storage:5> versions not found for apps: minio
unit-kfp-api-0: 08:03:26 ERROR unit.kfp-api/0.juju-log Failed to handle <PebbleReadyEvent via KfpApiOperator/on/apiserver_pebble_ready[36]> with error: List of <ops.model.Relation object-storage:5> versions not found for apps: minio

INFO     pytest_operator.plugin:plugin.py:855 Forgetting main...
=============================== warnings summary ===============================
tests/integration/test_bundle.py::test_build_and_deploy
tests/integration/test_bundle.py::test_build_and_deploy
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:218: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._drain_lock = asyncio.Lock(loop=loop)

tests/integration/test_bundle.py: 30 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:977: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.sleep(self.ping_interval, loop=self.loop)

tests/integration/test_bundle.py: 407 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:911: DeprecationWarning: 'with (yield from lock)' is deprecated use 'async with lock' instead
    with (yield from self._drain_lock):

tests/integration/test_bundle.py: 378 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:416: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait(

tests/integration/test_bundle.py: 28 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:988: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_bundle.py::test_build_and_deploy
tests/integration/test_bundle.py::test_build_and_deploy
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:532: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_bundle.py::test_build_and_deploy
tests/integration/test_bundle.py::test_build_and_deploy
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:554: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_bundle.py::test_build_and_deploy
tests/integration/test_bundle.py::test_build_and_deploy
  /home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/websockets/protocol.py:1077: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/integration/test_bundle.py::test_build_and_deploy - juju.errors....
================= 1 failed, 851 warnings in 292.25s (0:04:52) ==================
bundle-integration: exit 1 (293.06 seconds) /home/runner/work/kfp-operators/kfp-operators> pytest -vv --tb=native -s --model kubeflow --destructive-mode /home/runner/work/kfp-operators/kfp-operators/tests//integration pid=16225
  bundle-integration: FAIL code 1 (307.24=setup[14.18]+cmd[293.06] seconds)
  evaluation failed :( (307.30 seconds)
Error: Process completed with exit code 1.

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5520.

This message was autogenerated

[misohu]

The bundle integration test is using file tests/integration/data/kfp_against_latest_edge.yaml with list of charms which should be deployed. This list had pinned versions to latest/stable. I changed the versions according to versions in bundle-kubeflow release 1.7.