Upgrade vector (#378)

* adapt to vector upgrade

* changelog

* bump operator-rs

* stackable image version

* fix tests

* fix linter

* pr review

Author: maltesander

CHANGELOG.md

@@ -8,6 +8,12 @@ All notable changes to this project will be documented in this file.
 
 - Default stackableVersion to operator version ([#381]).
 
+### Changed
+
+- `vector` `0.26.0` -> `0.31.0` ([#378]).
+- `operator-rs` `0.44.0` -> `0.45.1` ([#381]).
+
+[#378]: https://github.com/stackabletech/hdfs-operator/pull/378
 [#381]: https://github.com/stackabletech/hdfs-operator/pull/381
 
 ## [23.7.0] - 2023-07-14

tests/kuttl-test.yaml.jinja2

@@ -9,3 +9,19 @@ testDirs:
 startKIND: false
 suppress: ["events"]
 parallel: 2
+
+# The timeout (in seconds) is used when namespaces are created or
+# deleted, and, if not overridden, in TestSteps, TestAsserts, and
+# Commands. If not set, the timeout is 30 seconds by default.
+#
+# The deletion of a namespace can take a while until all resources are
+# gracefully shut down. If the timeout is reached in the meantime, even
+# a successful test case is considered a failure.
+#
+# For instance, the termination grace period of the Vector aggregator in
+# the logging tests is set to 60 seconds. If there are logs entries
+# which could not be forwarded yet to the external aggregator defined in
+# the VECTOR_AGGREGATOR environment variable, then the test aggregator
+# uses this period of time by trying to forward the events. In this
+# case, deleting a namespace with several Pods takes about 90 seconds.
+timeout: 120

tests/templates/kuttl/logging/01-install-hdfs-vector-aggregator.yaml

@@ -5,7 +5,7 @@ commands:
   - script: >-
       helm install hdfs-vector-aggregator vector
       --namespace $NAMESPACE
-      --version 0.20.1
+      --version 0.23.0
       --repo https://helm.vector.dev
       --values hdfs-vector-aggregator-values.yaml
 ---

tests/templates/kuttl/logging/hdfs-vector-aggregator-values.yaml.j2

@@ -192,7 +192,7 @@ customConfig:
       type: filter
       inputs: [vector]
       condition: |-
-        .timestamp == to_timestamp!(0) ||
+        .timestamp == from_unix_timestamp!(0) ||
         is_null(.level) ||
         is_null(.logger) ||
         is_null(.message)

tests/templates/kuttl/logging/test_log_aggregation.py

@@ -2,7 +2,7 @@
 import requests
 
 
-def check_processed_events():
+def check_sent_events():
     response = requests.post(
         'http://hdfs-vector-aggregator:8686/graphql',
         json={
@@ -12,8 +12,8 @@ def check_processed_events():
                         nodes {
                             componentId
                             metrics {
-                                processedEventsTotal {
-                                    processedEventsTotal
+                                sentEventsTotal {
+                                    sentEventsTotal
                                 }
                             }
                         }
@@ -30,18 +30,19 @@ def check_processed_events():
 
     transforms = result['data']['transforms']['nodes']
     for transform in transforms:
+        sentEvents = transform['metrics']['sentEventsTotal']
         componentId = transform['componentId']
-        processedEvents = transform['metrics']['processedEventsTotal']
+
         if componentId == 'filteredInvalidEvents':
-            assert processedEvents is None or \
-                processedEvents['processedEventsTotal'] == 0, \
-                'Invalid log events were processed.'
+            assert sentEvents is None or \
+                   sentEvents['sentEventsTotal'] == 0, \
+                   'Invalid log events were sent.'
         else:
-            assert processedEvents is not None and \
-                processedEvents['processedEventsTotal'] > 0, \
-                f'No events were processed in "{componentId}".'
+            assert sentEvents is not None and \
+                   sentEvents['sentEventsTotal'] > 0, \
+                   f'No events were sent in "{componentId}".'
 
 
 if __name__ == '__main__':
-    check_processed_events()
+    check_sent_events()
     print('Test successful!')