Skip to content

Kubernetes

kubectl

get context

[2019-05-22 10:02] jgebhardt@jd-laptop: ~ $ kubectl config get-contexts
CURRENT   NAME             CLUSTER          AUTHINFO         NAMESPACE
          k8s-int          k8s-int          k8s-int
*         k8s-jgebhardt    k8s-jgebhardt    k8s-jgebhardt
          minikube         minikube         minikube

switch context

[2019-05-22 10:02] jgebhardt@jd-laptop: ~ $ kubectl config use-context k8s-int
Switched to context "k8s-int".

switch namespace

[2020-02-08 14:07:01 (k8s.jd:default)] jgebhardt@jd-laptop: ~ $ kubectl config set-context --current --namespace=gitlab-managed-apps
Context "k8s.jd" modified.

use kubectl with evil loadbalancer config

kubectl --insecure-skip-tls-verify --kubeconfig=cluster1.conf get nodes
kubectl --insecure-skip-tls-verify --kubeconfig=cluster2.conf get nodes
kubectl --insecure-skip-tls-verify --kubeconfig=cluster3.conf get nodes

port forwarding

kubectl port-forward <pod> <port>

Issues encountered

Pod Networking failing

Recently Pods encountered difficulties to reach each other.
Viewing logs you could see that these containers couldn't reach the api either.
As it turns out calico on the minions somehow dropped it IP.
By restarting docker on these nodes calico gets back to work.

Strange events
ubuntu@jump01:~$ kl get events --all-namespaces
NAMESPACE     LAST SEEN   FIRST SEEN   COUNT     NAME                                         KIND          SUBOBJECT                          TYPE      REASON                    SOURCE                               MESSAGE
[...]
monitoring    16s         16s          1         alertmanager-main-0.1560323e91212d48         Pod           spec.containers{config-reloader}   Normal    Created                   kubelet, node-2-k8s   Created container
monitoring    15s         15s          1         alertmanager-main-0.1560323eb1eac58d         Pod           spec.containers{config-reloader}   Normal    Started                   kubelet, node-2-k8s   Started container
monitoring    10s         10s          1         alertmanager-main-0.1560323fed89810e         Pod           spec.containers{alertmanager}      Warning   Unhealthy                 kubelet, node-2-k8s   Liveness probe failed: Get http://10.233.93.11:9093/api/v1/status: dial tcp 10.233.93.11:9093: getsockopt: connection refused
monitoring    1s          11s          3         alertmanager-main-0.1560323face43dee         Pod           spec.containers{alertmanager}      Warning   Unhealthy                 kubelet, node-2-k8s   Readiness probe failed: Get http://10.233.93.11:9093/api/v1/status: dial tcp 10.233.93.11:9093: getsockopt: connection refused
Pods can not reach other
ubuntu@jump01:~$ kl logs -n monitoring prometheus-k8s-0 prometheus
[...]
level=error ts=2018-10-22T16:52:07.65126041Z caller=notifier.go:473 component=notifier alertmanager=http://10.233.1.197:9093/api/v1/alerts count=1 msg="Error sending alert" err="context deadline exceeded"
level=error ts=2018-10-22T16:52:07.651293742Z caller=notifier.go:473 component=notifier alertmanager=http://10.233.1.194:9093/api/v1/alerts count=1 msg="Error sending alert" err="Post http://10.233.1.194:9093/api/v1/alerts: dial tcp 10.233.1.194:9093: i/o timeout"
[...]
No more calico tunl address on minions
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-1-k8s ip a s dev tunl0
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.233.1.192/32 brd 10.233.1.192 scope global tunl0
    valid_lft forever preferred_lft forever
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-2-k8s ip a s dev tunl0
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-3-k8s ip a s dev tunl0
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
restart docker on affected nodes
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-2-k8s sudo systemctl restart docker
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-3-k8s sudo systemctl restart docker
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-2-k8s ip a s dev tunl0
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.233.93.13/32 brd 10.233.93.13 scope global tunl0
    valid_lft forever preferred_lft forever
ubuntu@jump01:~$ ssh -i .ssh/priv.pem node-3-k8s ip a s dev tunl0
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.233.97.70/32 brd 10.233.97.70 scope global tunl0
    valid_lft forever preferred_lft forever
ubuntu@jump01:~$

Node unable to send updates to apiserver

Recently I stumbled upon many failing api-requests. One node tried to send status updates but api refuses these updates:

log
Oct 25 10:16:09 node-2-k8s kubelet[12567]: E1025 10:16:09.461473   12567 kubelet_node_status.go:377] Error updating node status, will retry: failed to patch status "{\"status\":{\"$setElementOrder/addresses\":[{\"type\":\"InternalIP\"},{\"type\":\"InternalIP\"},{\"type\":\"Hostname\"}],\"$setElementOrder/conditions\":[{\"type\":\"OutOfDisk\"},{\"type\":\"MemoryPressure\"},{\"type\":\"DiskPressure\"},{\"type\":\"PIDPressure\"},{\"type\":\"Ready\"}],\"addresses\":[{\"address\":\"10.64.43.8\",\"type\":\"InternalIP\"},{\"address\":\"10.150.28.15\",\"type\":\"InternalIP\"}],\"conditions\":[{\"lastHeartbeatTime\":\"2018-10-25T10:16:09Z\",\"type\":\"OutOfDisk\"},{\"lastHeartbeatTime\":\"2018-10-25T10:16:09Z\",\"type\":\"MemoryPressure\"},{\"lastHeartbeatTime\":\"2018-10-25T10:16:09Z\",\"type\":\"DiskPressure\"},{\"lastHeartbeatTime\":\"2018-10-25T10:16:09Z\",\"type\":\"PIDPressure\"},{\"lastHeartbeatTime\":\"2018-10-25T10:16:09Z\",\"type\":\"Ready\"}]}}" for node "node-2-k8s": Node "node-2-k8s" is invalid: status.addresses[1]: Duplicate value: core.NodeAddress{Type:"InternalIP", Address:"10.150.28.15"}
Oct 25 10:16:09 node-2-k8s kubelet[12567]: E1025 10:16:09.461646   12567 kubelet_node_status.go:377] Error updating node status, will retry: error getting node "node-2-k8s": Get https://10.150.28.9:6443/api/v1/nodes/node-2-k8s?timeout=10s: write tcp 10.150.28.15:45940->10.150.28.9:6443: use of closed network connection
Oct 25 10:16:09 node-2-k8s kubelet[12567]: E1025 10:16:09.461665   12567 kubelet_node_status.go:366] Unable to update node status: update node status exceeds retry count
Oct 25 10:16:09 node-2-k8s kubelet[12567]: W1025 10:16:09.461703   12567 reflector.go:341] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Unexpected watch close - watch lasted less than a second and no items received
Oct 25 10:16:09 node-2-k8s kubelet[12567]: W1025 10:16:09.461709   12567 reflector.go:341] k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: watch of *v1.Node ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: Unexpected watch close - watch lasted less than a second and no items received
Oct 25 10:16:09 node-2-k8s kubelet[12567]: W1025 10:16:09.461764   12567 reflector.go:341] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: watch of *v1.Service ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Unexpected watch close - watch lasted less than a second and no items received

This node have two interfaces. It semse that the apiserver gets confused about this: GitHub Issue #54492

By adding the --node-ip flag the problem vanished...

PV and PVC hanging in Terminate state forever

There might be left over finalizers. By removing them PV/PVC gets deleted immediately. See this GitHub Issue #69697

remove finalizer
kubectl edit pv <pvname>
# remove section containing "finalizer"

Renew Certificates prior to 1.13

Backup your config

tar -cvf kube.conf.tar /etc/kubernetes/

Renew all certificates

This will remove all certificates and all files for authentification

mv /etc/kubernetes/admin.conf{,.old}
mv /etc/kubernetes/controller-manager.conf{,.old}
mv /etc/kubernetes/kubelet.conf{,.old}
mv /etc/kubernetes/scheduler.conf{,.old}

find /etc/kubernetes/pki/ -mindepth 1 -type f -not -iname "*ca*" -and -not -iname "*sa*" -print -delete
Shutdown all Containers and kubelet and reissue all certificates
systemctl stop kubelet; systemctl restart docker;

kubeadm init phase certs all --apiserver-advertise-address 10.42.0.15 --apiserver-cert-extra-sans 10.42.0.11,10.42.0.26,10.42.0.32 --ignore-preflight-errors=all --node-name $HOSTNAME
Renew etcd certificates

For some reason etcd-certificates get issued the wrong way. So reissue them...

find /etc/kubernetes/pki/etcd/ -mindepth 1 -type f -not -iname "*ca*" -and -not -iname "*sa*" -print -delete
kubeadm init phase certs etcd-peers  --ignore-preflight-errors=all --node-name $HOSTNAME
systemctl stop kubelet; systemctl restart docker; systemctl start kubelet;
Restore previous config

Configuration get messed up. Restore your backed up manifests and restart kubelet

tar -C / -xvf /root/kube.conf.tar etc/kubernetes/manifests/
systemctl restart kubelet

Pin packages update and reboot
cat > /etc/apt/preferences.d/kubelet <EOF
Package: kube*
Pin: version 1.12.7*
Pin-Priority: 1000
EOF

apt update; apt -y dist-upgrade; reboot

move cni from weave to calico

Remove weave

kubectl delete -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Update pod cidr

kubeadm init phase control-plane controller-manager --pod-network-cidr=192.168.0.0/16

Install calico

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
helm repo add projectcalico https://docs.projectcalico.org/charts
helm install calico projectcalico/tigera-operator --version v3.20.0

If you want to use another cidr than calico's default. Change it in its DaemonSet

...
- name: CALICO_IPV4POOL_CIDR
value: "<your_podCIDR>"
...

Last update: August 5, 2021

Comments