Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Set the kubelet --resolv-conf flag conditionally at kubeadm init time #845

Closed
luxas opened this issue May 22, 2018 · 5 comments · Fixed by kubernetes/kubernetes#64665
Closed

Set the kubelet --resolv-conf flag conditionally at kubeadm init time #845

luxas opened this issue May 22, 2018 · 5 comments · Fixed by kubernetes/kubernetes#64665
Assignees
@stealthybox
Labels
area/releasing area/upgrades lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor.
Milestone
v1.11

Comments

@luxas
Copy link
Member

luxas commented May 22, 2018

When kubernetes/kubernetes#63887 merges, we should set the kubelet's --resolv-conf flag to /run/systemd/resolve/resolv.conf conditionally at kubeadm init time, in cmd/kubeadm/app/phases/kubelet/flags.go.

This instead of doing it statically in the deb/rpm packages.
cc @randomvariable @detiber @stealthybox @kad
xref related issue: #822
xref related PR: kubernetes/kubernetes#63632
@luxas luxas added this to the v1.11 milestone May 22, 2018
This was referenced May 22, 2018
Closed
Closed
@luxas
Copy link
Member Author

luxas commented May 23, 2018

@randomvariable has volunteered to start working on this in a day or two. Please let us know here when you do 👍

@randomvariable
Copy link
Member

Was going to pick this up, but notice you have #847, but we also want run time detection, or sensible defaulting for newer Ubuntu's don't we?

@luxas
Copy link
Member Author

luxas commented May 28, 2018

@randomvariable This should be done automatically. If systemd-resolved is used, the kubelet's --resolv-conf argument should be set by kubeadm so we don't break anyone. #847 basically means the user could do it him/herself, and even override kubeadm's defaulting for --resolv-conf, but doesn't mean we should automatically set this flag. Does that make sense to you?

@luxas luxas mentioned this issue May 29, 2018
Closed
@stealthybox
Copy link
Member

/assign

@stealthybox
Copy link
Member

We plan to get this into 1.11 before the weekend.
-- discussed in Kubeadm Office Hours - Wed May 30

@luxas luxas added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label May 30, 2018
stealthybox added a commit to stealthybox/kubernetes that referenced this issue Jun 3, 2018
@stealthybox
kubeadm: Set the kubelet --resolv-conf flag conditionally on init
b7df5cd 
Fixes kubernetes/kubeadm#845
@stealthybox stealthybox mentioned this issue Jun 3, 2018
Merged
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Jun 4, 2018
Merge pull request #64665 from stealthybox/feature/kubeadm_845-system…
86ae84b 
…d-resolved

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

kubeadm: Set the kubelet `--resolv-conf` flag conditionally on init

**What this PR does / why we need it**:
`kubeadm init` detects if systemd-resolved is running and configures the kubelet to use a working resolv.conf.
This patch also removes the warning message prompting manual user action for this configuration.

/area kubeadm
/area kubelet
/area dns
/kind bug
/priority important-soon

/sig cluster-lifecycle
/assign @timothysc

**Which issue(s) this PR fixes** 
Fixes kubernetes/kubeadm#845

**Special notes for your reviewer**:
See the difference in `KUBELET_KUBEADM_ARGS` when running with this build and enabling the resolved daemon on Ubuntu 17.10:
```bash
root@vagrant:/vagrant/bin# bash << EOF
systemctl start systemd-resolved
./845_kubeadm init |& tail -n5
cat /var/lib/kubelet/kubeadm-flags.env

./845_kubeadm reset --force |& tail -n2
systemctl stop systemd-resolved
echo nameserver 8.8.8.8 > /etc/resolv.conf

./845_kubeadm init |& tail -n5
cat /var/lib/kubelet/kubeadm-flags.env

EOF
You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.0.2.15:6443 --token 77q84j.0342evur7rrfrwwx --discovery-token-ca-cert-hash sha256:190040f9c3adf8410bc6766dac79f8679870190564e15e8f8d1704fafa03f678

KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni --resolv-conf=/run/systemd/resolve/resolv.conf
I0603 06:49:16.592482   14106 reset.go:276] [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
I0603 06:49:16.592858   14106 reset.go:290] [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.0.2.15:6443 --token 8mdart.gp67vq3nh9urq4z5 --discovery-token-ca-cert-hash sha256:da6b2e5841546eae134524b045e782f0dd91a6b53becc8d69c15d9eab9c88758

KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
```

**Release note**:
```release-note
`kubeadm init` detects if systemd-resolved is running and configures the kubelet to use a working resolv.conf.
```
wenjiaswe pushed a commit to wenjiaswe/kubernetes that referenced this issue Jun 19, 2018
@stealthybox @wenjiaswe
kubeadm: Set the kubelet --resolv-conf flag conditionally on init
c219587 
Fixes kubernetes/kubeadm#845
This was referenced Sep 29, 2018
Closed
Closed
@bhack bhack mentioned this issue Oct 16, 2018
Closed
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
4490add 
This problem occurs because systems using systemd-resolved copy
127.0.0.53 from the host's /etc/resolv.conf.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
bf2d802 
This problem occurs because systems using systemd-resolved copy
127.0.0.53 from the host's /etc/resolv.conf.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
3b39821 
This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.

Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.
@vannrt vannrt mentioned this issue Dec 7, 2018
Open
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
f48df82 
This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.

Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.

Without the fix, the kubedns and dnsmasq containers would copy the host's `/etc/resolv.conf`:
```
\# This file is managed by man:systemd-resolved(8). Do not edit.
\#
\# This is a dynamic resolv.conf file for connecting local clients to the
\# internal DNS stub resolver of systemd-resolved. This file lists all
\# configured search domains.
\#
\# Run "systemd-resolve --status" to see details about the uplink DNS servers
\# currently in use.
\#
\# Third party programs must not access this file directly, but only through the
\# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
\# replace this symlink by a static file or a different symlink.
\#
\# See man:systemd-resolved.service(8) for details about the supported modes of
\# operation for /etc/resolv.conf.

nameserver 127.0.0.53
search platform9.sys
```

After the fix:
```
\# This file is managed by man:systemd-resolved(8). Do not edit.
\#
\# This is a dynamic resolv.conf file for connecting local clients directly to
\# all known uplink DNS servers. This file lists all configured search domains.
\#
\# Third party programs must not access this file directly, but only through the
\# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
\# replace this symlink by a static file or a different symlink.
\#
\# See man:systemd-resolved.service(8) for details about the supported modes of
\# operation for /etc/resolv.conf.

nameserver 10.105.16.2
nameserver 10.105.16.4
search platform9.sys
```
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@stealthybox stealthybox

Labels
area/releasing area/upgrades lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor.
Projects
None yet
Milestone
v1.11
Development

Successfully merging a pull request may close this issue.

kubeadm: Set the kubelet --resolv-conf flag conditionally on init stealthybox/kubernetes
3 participants
@randomvariable @stealthybox @luxas