This repository was archived by the owner on Nov 9, 2021. It is now read-only.
Initialize Master failing #33
Comments
|
Hi @jmreicha. This is a warning from Ansible. We'll work on this. But, your cluster should be fine. |
|
@chris-short Thanks for the quick response. Unfortunately none of the containers have come up on the master. There is an error One thing I have found is that some others are having issues with k8s 1.10.3 so I think I might make a PR for specifying the version, if I can get it working with an older version. |
|
I can confirm that the deployment of a I have tested the deployment of kubernetes multiple times to day, on fresh images and I get the same error as jmreicha. A couple of week before the deployment went successful with the same setup. I expect that upstream changes in OS running on Ansible host:Ansible Version (ansible --version):Uploaded logs showing errors(rak8s/.log/ansible.log):Kubelet loggingOn the master: On a node: Raspberry Pi Hardware Version:3B+ Raspberry Pi OS & Version (cat /etc/os-release):Detailed description of the issue:I did run the The ansible playbook exited with a couple of messages. Only one is really fatal, the same as jmreicha got: The other messages are just warning:
Also the other message jmreicha got is a harmless warning:
There are no images pulled on the master and the nodes: I expect that the issue is caused due to a upstream change. This
During the last few months the versions of these components frequently changed: If we want to guarantee a stable I will prepare a pull request for fixing components to a specific version. |
|
FWIW I have it working with Kubernetes 1.10.2 and Docker 18.04. |
|
Quote jmreicha: jmreicha, thanks for the update. If have fixed the versions of roles/kubeadm/tasks/main.yml roles/master/tasks/main.yml Then I removed the what was there of This leaves use nearly fresh nodes. After a reboot of the nodes, I re-run the This time the Kubelet logging: We should https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aissue+ for related issues. to be continued... |
|
Did you reinstall Docker 18.05? I had to explicitly install 18.04. I can post my modification I found you want, it is similar to how you did the Kubernetes version. |
|
Thanks for your feedback jmreicha! The following To fix the After I run The playbook install is successful, but we are not yet there: pi@master:~ $ kubectl get nodes pi@master:~ $ kubectl version pi@master:~ $ sudo docker ps Now Unfortunately the cluster doesn't work. The master and nodes hang from time to time and containers are re-restarted. Needs more investigation.... |
|
That kernel version looks suspicious. What OS and kernel are you using? I used the Raspbian Stretch lite image with the 4.14 kernel. |
|
I used also raspbian stretch lite (release date 18-04-2018, kernel 4.14). |
|
Probably not the kernel then. It looks like Weave is having problems? Are you rebooting the nodes after your |
|
Successful deployment with kubernetes 1.10.2 & docker 18.04 worked, but after some minutes the API is no longer reachable. Note: I tried several combinations of Kubernetes (1.10, 1.10.1 1.10.2 and 1.10.3) in combination with Docker (18-04, 18-03, 18-02, 18-01, 17-12). None relay worked. kubernetes 1.10.2 and 18-04 looks like the best, because it is the last know good working combination. Cleanup ( Re-run with:
Successful, but some minutes later the API is unreachable: Kubelet logging master: I hope someone can find a cause in the logging. |
|
I rebooted the.nodes after "kubeadm reset* and after I removed the docker
and kubernetes packages.
Op di 29 mei 2018 22:40 schreef Josh Reichardt <notifications@github.com>:
… Probably not the kernel then. It looks like Weave is having problems? Are
you rebooting the nodes after your kubeadm reset?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#33 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AJoVDzvVWurtxdljZ7Nf1DJrPle2sPr5ks5t3bIygaJpZM4UQdPY>
.
|
|
That's a bummer, I haven't run into that error yet and I don't really have any other ideas off the top of my head, unless maybe Weave is the problem? Might try swapping it for Flannel. |
|
There might be some good tidbits in this issue geerlingguy/raspberry-pi-dramble#100 (comment) |
|
I like to test with a combination of versions that is known to be running successful. So if anyone has a rak8s cluster that is still running fine, I would like to know the versions of:
Please run the commands below on your Ansible host and post the output in this issue. Thanks a lot! |
|
Master:
Workers:
|
|
We are also working on building a basic support matrix over here if you want to help. |
|
so i can confirm on Debian jessie it doesnt works |
|
For the last few weeks a did a lot of testing on this. I was not able to get a kubernetes cluster version 1.10.1, 1.102 or 1.10.3 running on my 3 raspberry 3B+ nodes in combination with docker-ce 18-04 or 18-03. It always failed on Many others reported similar issues on Raspberry with Kubernetes 1.10.x at https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aissue+raspberry , so I decided to go back to a version that was reported to work okay. When I tried Kubernetes I now have added versioning to my rak8s cluster. I am able to configure the versions of docker-ce, kubeadm, kubectl, kubelet, kubernetes images and flannel in the inventory file. It is useful for:
My Inventory: Furthermore I have created a I will do a pull request, so everyone can enjoy a working cluster again. |
|
@tedsluis Very nice. Just a heads up on the Unfortunately I have no idea how to fix it. |
|
@jmreicha: yeah, I read about the etcd crypto issue on ARM. In one of my earlier post I requested for a combination of versions that still worked. You shared your working setup (thanks): Quote jmreicha: I was not able to get this combination up and running. Can you reproduce the installation from a fresh raspbian image and share your installation steps and logs, please? This would be very useful. |
This appears to be an error from Docker - it cannot navigate its own internal structures. In similar situations I have been reduced to deleting the entire Docker directory and starting again. |
|
@bboreham: Thanks for your remarks! I will keep that in mind. |
|
@tedsluis I have taken a similar approach in pinning different versions. I basically forked the repo so that I could manager RPi and Rock64's and ended up changing a bunch of other stuff. It's not on Github yet but should be soon 😄 Just rebuilt the cluster this morning with Docker 18.04 and Kubernetes 1.10.2 and has been stable. The main difference in my setup is that the workers are Rock64 boards instead of RPi. Master (RPi) Workers (Rock64) The only thing I haven't been able to get working yet is Weave fastdp (haven't tried Flannel). Apparently fastdp needs a kernel module I don't have and I haven't been brave enough to try a different kernel version yet. |
|
(A Weave Net makntainer writes) |
|
@bboreham Yep it is working well, just no vxlan yet. |
|
@jmreicha: Thanks for your update. I am curious about the version of kubeadm at the time you deployed the cluster. Could that be 1.9.x? Can you redeploy your cluster on a fresh raspbian image with kubeadm 1.10.x and 1.9.7 and then test whether it is still running? I am able to deploy a cluster with kubeadm and kubelet all on version 1.9.7 in combination with kubernetes images of version 1.101 and 1.10.2. Higher versions of kubeadm and kubelet will cause issues. Of course if I upgrade kubeadm to 1.10.x after the cluster deployment, the cluster will keep running. @bboreham: Thanks for your note. I will keep that in mind when I return back to flannel. |
|
@chris-short For what it's worth, I merged in @tedsluis changes and I was able to stand up a cluster using kubeadm/kubelet version 1.9.7 and docker version 18.04. flannel version 0.10.0. |
|
If the merged changes work I'm okay to close 😀 |
Merged
|
FYI for anyone thats interested, a kernel update has been released that should get vxlan working with RPi3B+ ARM-v7+. you can get it by doing an Details over at: raspberrypi/linux#2580 |
# for free
to subscribe to this conversation on GitHub.
Already have an account?
#.
OS running on Ansible host:
Ubuntu 16.04Ansible Version (
ansible --version):2.5.3Uploaded logs showing errors(
rak8s/.log/ansible.log)Raspberry Pi Hardware Version:
RPi 3B+Raspberry Pi OS & Version (
cat /etc/os-release):Raspbian GNU/Linux 9 (stretch)Detailed description of the issue:
Receive the above logs on a fresh install on the master. I haven't played around with it yet but figured I would let you know.
The text was updated successfully, but these errors were encountered: