Add yorkie-analytics Helm chart with Starrocks and Kafka (#1161)

* Add yorkie-analytics Helm chart with Starrocks

* Add Kafka yorkie-analytics Helm chart

* Enhance Yorkie deployment and Docker Compose for Starrocks integration

* Update yorkie-analytics chart to adjust CPU resource requests

* Add Kafka topic configuration for partitions and replication factor

---------

Co-authored-by: Youngteac Hong <susukang98@gmail.com>

.github/workflows/chart-release.yml

@@ -32,6 +32,7 @@ jobs:
           helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
           helm repo add grafana https://grafana.github.io/helm-charts
           helm repo add bitnami https://charts.bitnami.com/bitnami
+          helm repo add starrocks https://starrocks.github.io/starrocks-kubernetes-operator
           helm repo update
       - name: Run chart-releaser
         uses: helm/chart-releaser-action@v1.6.0

build/charts/README.md

@@ -7,3 +7,4 @@ This directory contains the Helm charts for Yorkie.
 - [yorkie-cluster](./yorkie-cluster): Helm chart for Yorkie cluster
 - [yorkie-monitoring](./yorkie-monitoring): Helm chart for Yorkie cluster monitoring system with Prometheus and Grafana
 - [yorkie-argocd](./yorkie-argocd): Helm chart for Yorkie cluster continuous delivery system with ArgoCD
+- [yorkie-analytics](./yorkie-analytics): Helm chart for Yorkie cluster analytics system with Kafka and StarRocks

build/charts/yorkie-analytics/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

build/charts/yorkie-analytics/Chart.lock

@@ -0,0 +1,9 @@
+dependencies:
+- name: kafka
+  repository: oci://registry-1.docker.io/bitnamicharts
+  version: 31.3.1
+- name: kube-starrocks
+  repository: https://starrocks.github.io/starrocks-kubernetes-operator
+  version: 1.9.10
+digest: sha256:602c748c6e5713880b3547ed0b0a9da2ed32b001b0dbabd97e372f305fadce8e
+generated: "2025-02-19T16:28:18.089273+09:00"

build/charts/yorkie-analytics/Chart.yaml

@@ -0,0 +1,34 @@
+apiVersion: v2
+name: yorkie-analytics
+description: yorkie-analytics provides analytics system with kafka and starrocks for analyzing usage of yorkie cluster
+type: application
+icon: https://raw.githubusercontent.com/yorkie-team/yorkie-team.github.io/main/public/favicon-512x512.png
+maintainers:
+  - name: hackerwins
+    email: susukang98@gmail.com
+  - name: krapie
+    email: krapi0314@gmail.com
+
+sources:
+  - https://github.com/yorkie-team/yorkie
+version: 0.6.0
+appVersion: "0.6.0"
+kubeVersion: ">=1.24.0-0"
+
+keywords:
+  - yorkie
+  - cluster
+  - kubernetes
+  - kafka
+  - starrocks
+
+# Kube Kafka and Starrocks stack dependencies
+dependencies:
+  - name: kafka
+    version: 31.3.1
+    repository: oci://registry-1.docker.io/bitnamicharts
+    condition: yorkie-analytics.kafka.enabled
+  - name: kube-starrocks
+    version: 1.9.10
+    repository: https://starrocks.github.io/starrocks-kubernetes-operator
+    condition: yorkie-analytics.starrocks.enabled

build/charts/yorkie-analytics/README.md

@@ -0,0 +1,68 @@
+# yorkie-analytics
+
+Installs the yorkie-analytics, which provides analytics system with Kafka and StarRocks for analyizing usage of yorkie cluster.
+
+## Prerequisites
+
+- Kubernetes 1.24+
+- Helm 3+
+
+## Get Helm Repository Info
+
+```bash
+helm repo add yorkie-team https://yorkie-team.github.io/yorkie/helm-charts
+helm repo update
+```
+
+_See [`helm repo`](https://helm.sh/docs/helm/helm_repo/) for command documentation._
+
+## Install Helm Chart
+
+```bash
+# Install yorkie analytics helm chart
+helm install [RELEASE_NAME] yorkie-team/yorkie-analytics -n analytics --create-namespace
+```
+
+_See [configuration](#configuration) below for custom installation_
+
+_See [`helm install`](https://helm.sh/docs/helm/helm_install/) for command documentation._
+
+## Dependencies
+
+By default this chart installs additional, dependent charts:
+
+- [kube-starrocks](https://starrocks.github.io/starrocks-kubernetes-operator)
+- [bitnami/kafka](https://github.com/bitnami/charts/tree/main/bitnami/kafka)
+
+_See [`helm dependency`](https://helm.sh/docs/helm/helm_dependency/) for command documentation._
+
+## Uninstall Helm Chart
+
+```bash
+helm uninstall [RELEASE_NAME] -n analytics
+```
+
+This removes all the Kubernetes components associated with the chart and deletes the release.
+
+_See [`helm uninstall`](https://helm.sh/docs/helm/helm_uninstall/) for command documentation._
+
+## Upgrading Chart
+
+```bash
+helm upgrade [RELEASE_NAME] yorkie-team/yorkie-analytics -n analytics
+```
+
+With Helm v3, CRDs created by this chart are not updated by default and should be manually updated.
+Consult also the [Helm Documentation on CRDs](https://helm.sh/docs/chart_best_practices/custom_resource_definitions).
+
+_See [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/) for command documentation._
+
+## Configuration
+
+See [Customizing the Chart Before Installing](#customizing-the-chart-before-installing) below for configuration options. To see all configurable options with detailed comments:
+
+```console
+helm show values yorkie-team/yorkie-analytics
+```
+
+You may also `helm show values` on this chart's [dependencies](#dependencies) for additional options.

build/charts/yorkie-analytics/templates/NOTES.txt

@@ -0,0 +1,9 @@
+--- Install Complete ---
+{{ .Release.Name }} successfully installed!
+
+For next steps, follow:
+  $ curl https://github.com/yorkie-team/yorkie/tree/main/charts/yorkie-analytics
+
+To learn more about the release, try:
+  $ helm status {{ .Release.Name }}
+  $ helm get all {{ .Release.Name }}

build/charts/yorkie-analytics/templates/kafka/configmap.yaml

@@ -0,0 +1,24 @@
+{{- if index .Values "yorkie-analytics" "kafka" "enabled" }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Values.name }}-init-kafka-topics-script
+  namespace: {{ .Values.namespace }}
+  labels:
+    app.kubernetes.io/component: kafka
+    app.kubernetes.io/part-of: yorkie-analytics
+data:
+  init-kafka-topics.sh: |
+    #!/bin/bash
+
+    KAFKA_HOST={{ .Values.name }}-kafka.{{ .Values.namespace }}.svc.cluster.local:9092
+    
+    echo -e 'Waiting for Kafka to be ready...'
+    kafka-topics.sh --bootstrap-server $KAFKA_HOST --list
+
+    echo -e 'Creating kafka topics'
+    kafka-topics.sh --bootstrap-server $KAFKA_HOST --create --if-not-exists --topic {{ index .Values "yorkie-analytics" "kafka" "topic" "name" }} --partitions {{ index .Values "yorkie-analytics" "kafka" "topic" "partitions" }} --replication-factor {{ index .Values "yorkie-analytics" "kafka" "topic" "replicationFactor" }}
+
+    echo -e 'Successfully created the following topics:'
+    kafka-topics.sh --bootstrap-server $KAFKA_HOST --list
+{{- end }}

build/charts/yorkie-analytics/templates/kafka/job.yaml

@@ -0,0 +1,34 @@
+{{- if index .Values "yorkie-analytics" "kafka" "enabled" }}
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: {{ .Values.name }}-init-kafka-topics-job
+  namespace: {{ .Values.namespace }}
+  labels:
+    app.kubernetes.io/component: kafka
+    app.kubernetes.io/part-of: yorkie-analytics
+  annotations:
+    "helm.sh/hook": post-install,post-upgrade
+    "helm.sh/hook-delete-policy": before-hook-creation
+spec:
+  template:
+    spec:
+      serviceAccountName: starrocks
+      restartPolicy: OnFailure
+      initContainers:
+        - name: wait-for-kafka
+          image: bitnami/kubectl
+          command: [ "kubectl", "wait", "pod/{{ $.Values.name }}-kafka-controller-0", "--for", "condition=Ready", "--namespace", "{{ .Values.namespace }}", "--timeout", "300s" ]
+      containers:
+        - name: init-kafka-topics
+          image: {{ index .Values "yorkie-analytics" "kafka" "image" "repository" }}:{{ index .Values "yorkie-analytics" "kafka" "image" "tag" }}
+          command: [ "/bin/bash", "/etc/config/init-kafka-topics.sh" ]
+          volumeMounts:
+          - name: init-kafka-topics-script
+            mountPath: /etc/config
+      volumes:
+      - name: init-kafka-topics-script
+        configMap:
+          name: {{ .Values.name }}-init-kafka-topics-script
+  backoffLimit: 10
+{{- end }}

build/charts/yorkie-analytics/templates/starrocks/configmap.yaml

@@ -0,0 +1,75 @@
+{{- if index .Values "yorkie-analytics" "starrocks" "enabled" }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Values.name }}-init-starrocks-database-script
+  namespace: {{ .Values.namespace }}
+  labels:
+    app.kubernetes.io/component: starrocks
+    app.kubernetes.io/part-of: yorkie-analytics
+data:
+  init-user-events-db.sql: |
+    CREATE DATABASE IF NOT EXISTS yorkie;
+
+    USE yorkie;
+
+    CREATE TABLE user_events (
+        user_id VARCHAR(64),
+        timestamp DATETIME, 
+        event_type VARCHAR(32),
+        project_id VARCHAR(64),
+        user_agent VARCHAR(32)
+    ) ENGINE = OLAP  
+    DUPLICATE KEY(user_id)  
+    DISTRIBUTED BY HASH(user_id) BUCKETS 10
+    PROPERTIES (  
+        "replication_num" = "1"  
+    );
+
+  init-routine-load.sql: |
+    CREATE ROUTINE LOAD yorkie.events ON user_events
+    PROPERTIES
+    (
+        "format" = "JSON",
+        "desired_concurrent_number"="1"
+    )
+    FROM KAFKA
+    (   
+        "kafka_broker_list" = "{{ index .Values "yorkie-analytics" "starrocks" "routine-load" "kafka-broker-list" }}",
+        "kafka_topic" = "{{ index .Values "yorkie-analytics" "starrocks" "routine-load" "kafka-topic" }}",
+        "property.group.id" = "{{ index .Values "yorkie-analytics" "starrocks" "routine-load" "property-group-id" }}"
+    );
+
+  init-starrocks-database.sh: |
+    #!/bin/bash
+
+    STARROCKS_MYSQL_HOST=kube-starrocks-fe-service.{{ .Values.namespace }}.svc.cluster.local
+    
+    sleep 5s
+    echo -e 'Checking Starrocks status'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show frontends\G' | grep 'Alive: true' || echo -e 'Frontend is not ready'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show backends\G' | grep 'Alive: true' || echo -e 'Starrocks BE is not ready'
+
+
+    echo -e 'Creating Yorkie database, tables and routine load'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root < /etc/config/init-user-events-db.sql
+
+    echo -e 'Checking Yorkie database'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show databases\G'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show databases\G' | grep 'Database: yorkie' || echo -e 'Yorkie database not found'
+
+    echo -e 'Checking user_event table'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show tables from yorkie\G'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show tables from yorkie\G' | grep 'Tables_in_yorkie: user_events' || echo -e 'user_events table not found'
+
+
+    sleep 5s
+    echo -e 'Creating routine load'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root < /etc/config/init-routine-load.sql
+
+    sleep 10s
+    echo -e 'Checking event routine load'
+
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show routine load from yorkie\G'
+    mysql -h $STARROCKS_MYSQL_HOST -P 9030 -u root -e 'show routine load from yorkie\G' | grep -E "State: NEED_SCHEDULE|State: RUNNING" || echo -e 'Routine load is not running'
+{{- end }}

build/charts/yorkie-analytics/templates/starrocks/job.yaml

@@ -0,0 +1,42 @@
+{{- if index .Values "yorkie-analytics" "starrocks" "enabled" }}      
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: {{ .Values.name }}-init-starrocks-database-job
+  namespace: {{ .Values.namespace }}
+  labels:
+    app.kubernetes.io/component: starrocks
+    app.kubernetes.io/part-of: yorkie-analytics
+  annotations:
+    "helm.sh/hook": post-install,post-upgrade
+    "helm.sh/hook-delete-policy": before-hook-creation
+spec:
+  template:
+    spec:
+      serviceAccountName: starrocks
+      restartPolicy: OnFailure
+      initContainers:
+        - name: wait-for-starrocks-fe
+          image: bitnami/kubectl
+          command: [ "kubectl", "wait", "pod/kube-starrocks-fe-0", "--for", "condition=Ready", "--namespace", "{{ .Values.namespace }}", "--timeout", "300s" ]
+        - name: wait-for-starrocks-be
+          image: bitnami/kubectl
+          command: [ "kubectl", "wait", "pod/kube-starrocks-be-0", "--for", "condition=Ready", "--namespace", "{{ .Values.namespace }}", "--timeout", "300s" ]
+        {{- if index .Values "yorkie-analytics" "kafka" "enabled" }}
+        - name: wait-for-kafka
+          image: bitnami/kubectl
+          command: [ "kubectl", "wait", "pod/{{ $.Values.name }}-kafka-controller-0", "--for", "condition=Ready", "--namespace", "{{ .Values.namespace }}", "--timeout", "300s" ]
+        {{- end }}
+      containers:
+        - name: init-starrocks-database
+          image: {{ index .Values "yorkie-analytics" "starrocks" "image" "repository" }}:{{ index .Values "yorkie-analytics" "starrocks" "image" "tag" }}
+          command: [ "/bin/bash", "/etc/config/init-starrocks-database.sh" ]
+          volumeMounts:
+          - name: init-starrocks-database-script
+            mountPath: /etc/config
+      volumes:
+      - name: init-starrocks-database-script
+        configMap:
+          name: {{ .Values.name }}-init-starrocks-database-script
+  backoffLimit: 10
+{{- end }}