refactor yurtadm init/join and support to create high-availability OpenYurt cluster #926

windydayc · 2022-07-22T04:56:15Z

What type of PR is this?

/kind feature
/kind enhancement

What this PR does / why we need it:

At present, the installation of OpenYurt cluster is still a little complicated, there is a lack of a unified installation way that can simply and automatically install OpenYurt cluster.

In addition, currently the yurtadm command cannot cope with scenarios that requiring high availability. Therefore, it is necessary to provide a way to create high availability OpenYurt cluster.

Special notes for your reviewer:

/assign @rambohe-ch @Peeknut

openyurt-bot · 2022-07-22T04:56:19Z

@windydayc: GitHub didn't allow me to assign the following users: Peeknut.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

What type of PR is this?

/kind feature
/kind enhancement

What this PR does / why we need it:

At present, the installation of OpenYurt cluster is still a little complicated, there is a lack of a unified installation way that can simply and automatically install OpenYurt cluster.

In addition, currently the yurtadm command cannot cope with scenarios that requiring high availability. Therefore, it is necessary to provide a way to create high availability OpenYurt cluster.

Related issue: #856 #902

Special notes for your reviewer:

/assign @rambohe-ch @Peeknut

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openyurt-bot · 2022-07-22T05:27:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: windydayc
To complete the pull request process, please assign rambohe-ch
You can assign the PR to them by writing /assign @rambohe-ch in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2022-07-22T05:36:51Z

Codecov Report

Merging #926 (e93fbf7) into master (482fc1d) will increase coverage by 8.36%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #926      +/-   ##
==========================================
+ Coverage   35.30%   43.66%   +8.36%     
==========================================
  Files          76       83       +7     
  Lines        9913    11260    +1347     
==========================================
+ Hits         3500     4917    +1417     
+ Misses       6146     5908     -238     
- Partials      267      435     +168

Flag	Coverage Δ
unittests	`43.66% <ø> (+8.36%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/yurthub/server/nonresource.go	`31.08% <0.00%> (ø)`
pkg/controller/kubernetes/util/node/node.go	`82.35% <0.00%> (ø)`
pkg/yurthub/server/certificate.go	`0.00% <0.00%> (ø)`
pkg/controller/nodelifecycle/metrics.go	`100.00% <0.00%> (ø)`
pkg/controller/kubernetes/util/taints/taints.go	`82.08% <0.00%> (ø)`
pkg/yurthub/server/server.go	`0.00% <0.00%> (ø)`
...troller/nodelifecycle/node_lifecycle_controller.go	`56.15% <0.00%> (ø)`
pkg/yurthub/cachemanager/cache_manager.go	`68.22% <0.00%> (+0.07%)`	⬆️
pkg/yurthub/util/util.go	`21.37% <0.00%> (+0.86%)`	⬆️
pkg/yurtadm/util/edgenode/util.go	`9.02% <0.00%> (+7.35%)`	⬆️
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…enYurt cluster

rambohe-ch · 2022-07-25T07:46:58Z

config/yurtadm/cloudimage/README.md

@@ -0,0 +1,13 @@
+# Build an OpenYurt CloudImage
+
+`yurtadm init` is implemented by sealer, you can modify the kubefile to make your own openyurt cloudimage.


openyurt cloudimage --> openyurt cluster image

ok, solved.

rambohe-ch · 2022-07-25T07:49:08Z

config/yurtadm/cloudimage/README.md

+sealer build -t registry-1.docker.io/openyurt/openyurt-cluster:latest-k8s-1198 -f Kubefile .
+
+# push to dockerhub
+sealer push registry-1.docker.io/openyurt/openyurt-cluster:latest-k8s-1198


i think that common end user can not push openyurt-cluster image into OpenYurt dockerhub, because they can not get the password.

OK. It is just an example.

Solved.

rambohe-ch · 2022-07-25T08:19:31Z

pkg/yurtadm/README.md

+
+```bash
+# Initialize an OpenYurt cluster.
+yurtadm init --apiserver-advertise-address 192.168.152.131 --openyurt-version latest --passwd 1234


what's the mean of the address 192.168.152.131 and passwd 1234? where do end users can find these kind of information?

It is just an example. This is the IP address and SSH password of the node the user needs to init. Users can use yurtadm init -h to see the effect of these parameters.

i think we should make clear introduction in the tutorial for concrete ip address and password, if not, end user will not understand the meaning.

rambohe-ch · 2022-07-25T08:20:56Z

pkg/yurtadm/README.md

+
+### 3.2 Initialize the OpenYurt cluster
+Execute the following command as root account, no need to install container runtimes such as docker in advance. Docker will be automatically installed during the execution of `yurtadm init`.
+


I think we need to a image for explaining which kind of nodes and components are installed by yurtadm init and yurtadm join

docker is installed by sealer apply, so before yurtadm init, you don't need to install docker.
While sealer is not involved in yurtadm join, so before yurtadm join, you should install docker.

rambohe-ch · 2022-07-25T08:23:50Z

config/yurtadm/cloudimage/openyurt-latest/yamls/yurt-controller-manager.yaml

@@ -0,0 +1,148 @@
+apiVersion: v1


how about use yaml file in openyurtio/openyurt/config/setup directory? so we don't need to manage yaml file of openyurt here.

@rambohe-ch
At present, some parameters in yaml are not fixed, but are rendered by env in clusterfile, that is, the suffix .yaml.tmpl file. Later, if you need to add user-defined parameters in env, .yaml.tmpl files also need to be changed, so the file here (.yaml.tmpl ) may be somewhat different from that in the openyurtio/openyurt/config/setup directory.

see: Using Clusterfile to init a cluster - Env render support | sealer

rambohe-ch · 2022-07-25T08:26:52Z

config/yurtadm/cloudimage/openyurt-latest/flannel/init-kube.sh

@@ -0,0 +1,177 @@
+#!/bin/bash


init-kube.sh is used for initializing system for kubernetes, so it's not reasonable to put this file under flannel directory.

ok, solved.

rambohe-ch · 2022-07-25T08:28:19Z

config/yurtadm/cloudimage/openyurt-latest/flannel/kube-flannel.yml

+    }
+  net-conf.json: |
+    {
+      "Network": "10.244.0.0/16",


this network should be configured by end user

ok, solved.

rambohe-ch · 2022-07-25T08:32:45Z

config/yurtadm/cloudimage/openyurt-latest/flannel/kube-flannel.yml

+      serviceAccountName: flannel
+      initContainers:
+      - name: install-cni
+        image: quay.io/coreos/flannel:v0.14.0


we need to use the optimized flannel image, mentioned here: https://openyurt.io/docs/user-manuals/network/edge-pod-network/

btw: cni plugin host-local should use the optimized plugin, please pay attentation.

ok, solved.

rambohe-ch · 2022-07-25T08:33:57Z

config/yurtadm/cloudimage/openyurt-latest/flannel/kube-flannel.yml

+    # SELinux is unused in CaaSP
+    rule: 'RunAsAny'
+---
+kind: ClusterRole


cluster role should be suitable for the optimized flannel image.

ok, solved.

rambohe-ch · 2022-07-25T08:38:23Z

config/yurtadm/cloudimage/openyurt-latest/install-openyurt.sh

+## configure coredns
+kubectl apply -f manifests/coredns.yaml
+kubectl annotate svc kube-dns -n kube-system openyurt.io/topologyKeys='openyurt.io/nodepool'
+kubectl scale --replicas=0 deployment/coredns -n kube-system


maybe it's more reasonable to scale deployment/coredns before applying daemonset/coredns.

ok, solved.

rambohe-ch · 2022-07-25T08:40:05Z

config/yurtadm/cloudimage/openyurt-latest/Kubefile

+COPY install-openyurt.sh .
+RUN chmod 777 install-openyurt.sh
+
+CMD kubectl apply -f manifests/kube-flannel.yml


why not put kubectl apply -f manifests/kube-flannel.yml into install-openyurt.sh file?

ok, solved.

rambohe-ch · 2022-07-25T08:40:47Z

config/yurtadm/cloudimage/openyurt-latest/Kubefile

+FROM kubernetes:v1.19.8-alpine
+
+# flannel: https://github.com/sealerio/applications/tree/main/flannel
+COPY flannel/cni .


why put cni files under flannel directory?

rambohe-ch · 2022-07-25T08:42:13Z

config/yurtadm/cloudimage/openyurt-latest/Kubefile

+COPY flannel/init-kube.sh /scripts/
+COPY flannel/kube-flannel.yml manifests/
+
+COPY shell-plugin.yaml plugins


when do we apply this yaml file(shell-plugin.yaml)?

It is sealer's plugin: http://sealer.cool/docs/getting-started/plugin.html#plugin-type-list
Sealer will automatically execute all plugins under the plugins directory.

rambohe-ch · 2022-07-25T08:44:26Z

pkg/yurtadm/cmd/join/join.go

 	"github.com/openyurtio/openyurt/pkg/yurtadm/cmd/join/joindata"
 	yurtphase "github.com/openyurtio/openyurt/pkg/yurtadm/cmd/join/phases"
 	yurtconstants "github.com/openyurtio/openyurt/pkg/yurtadm/constants"
 	yurtadmutil "github.com/openyurtio/openyurt/pkg/yurtadm/util/kubernetes"
 )

 var (
+	joinExample = templates.Examples(i18n.T(`
+		# Join the edge node to cluster.
+		yurtadm join 192.168.152.131:6443 --token=zffaj3.a5vjzf09qn9ft3gt --node-type=edge --discovery-token-unsafe-skip-ca-verification --v=5


we should not use concrete ip address and token in the example

OK. I will change it to 1.2.3.4

rambohe-ch · 2022-07-25T08:51:08Z

pkg/yurtadm/cmd/yurtinit/init.go

 `
 )

 var (
+	initExample = templates.Examples(i18n.T(`
+		# Initialize an OpenYurt cluster.
+		yurtadm init --apiserver-advertise-address 192.168.152.131 --openyurt-version latest --passwd 1234


ok, solved.

rambohe-ch · 2022-07-25T08:57:15Z

config/yurtadm/cloudimage/openyurt-latest/flannel/init-kube.sh

+cgroupDriver: ${criDriver}
+cgroupsPerQOS: true
+clusterDNS:
+- 10.96.0.10


this ip address should be match with value that set by end user.

rambohe-ch · 2022-07-25T08:59:04Z

@windydayc please do not push --force when you fixed above comments.

rambohe-ch · 2022-07-25T09:33:09Z

@Peeknut PTAL

Peeknut · 2022-08-02T03:31:45Z

config/yurtadm/cluster-image/README.md

+cd openyurt-latest
+
+# build openyurt ClusterImage
+sealer build -t registry-1.docker.io/your_dockerhub_username/openyurt-cluster:latest-k8s-1198 -f Kubefile .


Maybe it is better to explain how to install sealer and the version of sealer.

Peeknut · 2022-08-02T03:38:01Z

config/yurtadm/cluster-image/openyurt-latest/install.sh

+kubectl annotate svc kube-dns -n kube-system openyurt.io/topologyKeys='openyurt.io/nodepool'
+
+## configure kube-proxy
+kubectl patch cm -n kube-system kube-proxy --patch '{"data": {"config.conf": "apiVersion: kubeproxy.config.k8s.io/v1alpha1\nbindAddress: 0.0.0.0\nfeatureGates:\n  EndpointSliceProxying: true\nbindAddressHardFail: false\nclusterCIDR: 100.64.0.0/10\nconfigSyncPeriod: 0s\nenableProfiling: false\nipvs:\n  excludeCIDRs:\n  - 10.103.97.2/32\n  minSyncPeriod: 0s\n  strictARP: false\nkind: KubeProxyConfiguration\nmode: ipvs\nudpIdleTimeout: 0s\nwinkernel:\n  enableDSR: false\nkubeconfig.conf:"}}'  && kubectl delete pod --selector k8s-app=kube-proxy -n kube-system


Here are many IP addresses, how to make sure these IP addresses are available to all users?

Peeknut · 2022-08-02T03:52:55Z

pkg/yurtadm/constants/constants.go

@@ -48,6 +48,9 @@ const (

 	DefaultOpenYurtImageRegistry = "registry.cn-hangzhou.aliyuncs.com/openyurt"
 	DefaultOpenYurtVersion       = "latest"
+	DefaultK8sVersion            = "1198" // default kubernetes 1.19.8


why not use 1.19.8? 1198 is unclear.

Peeknut · 2022-08-02T03:53:34Z

pkg/yurtadm/util/kubernetes/util.go

@@ -144,20 +144,35 @@ func CheckAndInstallKubelet(kubernetesResourceServer, clusterVersion string) err
 		klog.Infof("Cni dir %s already exist, skip install.", constants.KubeCniDir)
 		return nil
 	}
+
+	////download and install kubernetes-cni


Please delete notes.

Peeknut · 2022-08-02T03:56:55Z

pkg/yurtadm/util/kubernetes/util.go

 	//download and install kubernetes-cni
-	cniUrl := fmt.Sprintf(constants.CniUrlFormat, constants.KubeCniVersion, runtime.GOARCH, constants.KubeCniVersion)
-	savePath := fmt.Sprintf("%s/cni-plugins-linux-%s-%s.tgz", constants.TmpDownloadDir, runtime.GOARCH, constants.KubeCniVersion)
+	cniUrl := "https://github.com/openyurtio/openyurt/releases/download/v0.7.0/openyurt-cni-0.8.7-0.x86_64.rpm"


If using constants, it is better to define in const.
If configuration is supported, it is better to use fmt.Sprintf.

@Peeknut
openyurt-cni-0.8.7-0.x86_64.rpm(https://github.com/openyurtio/openyurt/releases/download/v0.7.0/openyurt-cni-0.8.7-0.x86_64.rpm) is only seen on https://github.com/openyurtio/openyurt/releases, and it has not changed.
It seems that it has little to do with the openyurt version.
Which one should be constants here? It seems that the openyurt version cannot be obtained in the yurtadm join codes. Even if it can be obtained, if it is latest version, there is no link like https://github.com/openyurtio/openyurt/releases/download/`latest`/openyurt-cni-0.8.7-0.x86_64.rpm

I replaced it with cniUrl := fmt.Sprintf("%s", constants.OpenYurtCniUrl)

windydayc · 2022-08-04T02:11:11Z

For config/yurtadm/cluster-image/openyurt-latest/cni directory, I have replaced the content in openyurt-cni-0.8.7-0.x86_64.rpm in https://github.com/openyurtio/openyurt/releases

Reason see: https://openyurt.io/docs/user-manuals/network/edge-pod-network/#ipam-pod-ip-address-kept

windydayc · 2022-08-10T11:30:27Z

For yurtadm init, since the image of the latest version often changes, its configuration file may also change, so the cluster image which constructed by the latest version may not be available. Therefore, an image with a fixed version number is used here to prevent cluster imgae unavailability caused by image:latest changes.
For example, here use yurt-controller-manager:v0.7.0 ，not yurt-controller-manager:latest

windydayc · 2022-08-10T12:02:52Z

@Peeknut I have solved the above problems. Please have a look.

huiwq1990 · 2022-08-13T01:59:38Z

@windydayc I think it's not a good idea to maintian the cni binaries in openyurt repo, could you explain the reason?
I recommend:

the binaries directly download from https://github.com/containernetworking/plugins/releases/
or
after the k8s starup, use a daemonset put the binaries to cni directory, the yaml like:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: test-cni-tool
  name: test-cni-tool
spec:
  selector:
    matchLabels:
      app: test-cni-tool
  template:
    metadata:
      labels:
        app: test-cni-tool
    spec:
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: "Exists"
      containers:
      - name: myapp-container
        image: busybox:1.28
        command: ['sh', '-c', 'echo The app is running! && sleep 3600h']
      initContainers:
        - name: hostname
          image: huiwq1990/cni-plugins
          imagePullPolicy: Always
          command: ["/bin/sh"]
          args: ["-c", "cp /cni/* /opt/cni/bin/"]
          volumeMounts:
           - name: task-pv-storage
             mountPath: /opt/cni/bin
        - name: restart
          image: jpetazzo/nsenter
          command: ["/bin/sh"]
          args: ["-c", "/nsenter --mount=/proc/1/ns/mnt systemctl restart containerd"]
          securityContext:
            privileged: true
      volumes:
        - name: task-pv-storage
          hostPath:
            path: /opt/cni/bin
            type: Directory

windydayc · 2022-08-13T05:28:29Z

@windydayc I think it's not a good idea to maintian the cni binaries in openyurt repo, could you explain the reason? I recommend:

the binaries directly download from https://github.com/containernetworking/plugins/releases/
or
after the k8s starup, use a daemonset put the binaries to cni directory, the yaml like:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: test-cni-tool
  name: test-cni-tool
spec:
  selector:
    matchLabels:
      app: test-cni-tool
  template:
    metadata:
      labels:
        app: test-cni-tool
    spec:
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: "Exists"
      containers:
      - name: myapp-container
        image: busybox:1.28
        command: ['sh', '-c', 'echo The app is running! && sleep 3600h']
      initContainers:
        - name: hostname
          image: huiwq1990/cni-plugins
          imagePullPolicy: Always
          command: ["/bin/sh"]
          args: ["-c", "cp /cni/* /opt/cni/bin/"]
          volumeMounts:
           - name: task-pv-storage
             mountPath: /opt/cni/bin
        - name: restart
          image: jpetazzo/nsenter
          command: ["/bin/sh"]
          args: ["-c", "/nsenter --mount=/proc/1/ns/mnt systemctl restart containerd"]
          securityContext:
            privileged: true
      volumes:
        - name: task-pv-storage
          hostPath:
            path: /opt/cni/bin
            type: Directory

@huiwq1990

cni here is from openyurt-cni-0.8.7-0.x86_64.rpm in https://github.com/openyurtio/openyurt/releases, not directly download from https://github.com/containernetworking/plugins/releases/. Reason see: https://openyurt.io/docs/user-manuals/network/edge-pod-network/#ipam-pod-ip-address-kept
The config/yurtadm/cluster-image/openyurt-v0.7.0 dir here is just an example for user reference, users can customize their own cluster image.

…nYurt cluster

openyurt-bot assigned rambohe-ch Jul 22, 2022

openyurt-bot added kind/feature kind/feature kind/enhancement kind/enhancement labels Jul 22, 2022

openyurt-bot requested review from huangyuqi and kadisi July 22, 2022 04:56

openyurt-bot added the size/XXL label Jul 22, 2022

windydayc force-pushed the yurtadm branch from f1f0eca to 7f00ae4 Compare July 22, 2022 05:27

windydayc force-pushed the yurtadm branch 3 times, most recently from 84ac931 to 2291e7b Compare July 22, 2022 06:31

windydayc added 2 commits July 22, 2022 14:48

refactor yurtadm init/join and support to create high-availability Op…

d9b2b3e

…enYurt cluster

add yurtadm cloudimage

1507589

windydayc force-pushed the yurtadm branch from 2291e7b to 1507589 Compare July 22, 2022 06:56

rambohe-ch reviewed Jul 25, 2022

View reviewed changes

windydayc added a commit to windydayc/openyurt that referenced this pull request Jul 27, 2022

fix problems in openyurtio#926 code review

c7b1a9e

windydayc force-pushed the yurtadm branch 2 times, most recently from c7b1a9e to c3c3726 Compare July 28, 2022 02:55

Peeknut reviewed Aug 2, 2022

View reviewed changes

windydayc force-pushed the yurtadm branch from 17eeded to 09ae03d Compare August 10, 2022 06:25

windydayc mentioned this pull request Aug 10, 2022

[Question] how to build openyurt sealer baseimage #942

Closed

windydayc force-pushed the yurtadm branch from 09ae03d to 0802e20 Compare August 10, 2022 11:23

windydayc force-pushed the yurtadm branch 2 times, most recently from 2878046 to 0e12e94 Compare August 10, 2022 12:01

windydayc mentioned this pull request Aug 11, 2022

[feature request] The k8s version problem involved in yurtadm join #908

Closed

windydayc force-pushed the yurtadm branch 9 times, most recently from 7f0d3c3 to e93fbf7 Compare August 22, 2022 10:58

improve yurtadm init/join and support to create high-availability Ope…

e93fbf7

…nYurt cluster

windydayc closed this Aug 23, 2022

		@@ -0,0 +1,13 @@
		# Build an OpenYurt CloudImage

		`yurtadm init` is implemented by sealer, you can modify the kubefile to make your own openyurt cloudimage.


		### 3.2 Initialize the OpenYurt cluster
		Execute the following command as root account, no need to install container runtimes such as docker in advance. Docker will be automatically installed during the execution of `yurtadm init`.

refactor yurtadm init/join and support to create high-availability OpenYurt cluster #926

refactor yurtadm init/join and support to create high-availability OpenYurt cluster #926

Conversation

windydayc commented Jul 22, 2022 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Special notes for your reviewer:

openyurt-bot commented Jul 22, 2022

What type of PR is this?

What this PR does / why we need it:

Special notes for your reviewer:

openyurt-bot commented Jul 22, 2022

codecov bot commented Jul 22, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

windydayc Jul 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Jul 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rambohe-ch commented Jul 25, 2022

rambohe-ch commented Jul 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc Aug 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windydayc commented Aug 4, 2022 • edited Loading

windydayc commented Aug 10, 2022

windydayc commented Aug 10, 2022

huiwq1990 commented Aug 13, 2022

windydayc commented Aug 13, 2022 • edited Loading

windydayc commented Jul 22, 2022 •

edited

Loading

codecov bot commented Jul 22, 2022 •

edited

Loading

windydayc Jul 26, 2022 •

edited

Loading

windydayc Jul 25, 2022 •

edited

Loading

windydayc Jul 25, 2022 •

edited

Loading

windydayc Jul 26, 2022 •

edited

Loading

windydayc Jul 28, 2022 •

edited

Loading

windydayc Jul 28, 2022 •

edited

Loading

windydayc Jul 28, 2022 •

edited

Loading

windydayc Jul 28, 2022 •

edited

Loading

windydayc Jul 25, 2022 •

edited

Loading

windydayc Jul 25, 2022 •

edited

Loading

windydayc Aug 6, 2022 •

edited

Loading

windydayc commented Aug 4, 2022 •

edited

Loading

windydayc commented Aug 13, 2022 •

edited

Loading