Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
IN-DEPTH TECH
4 min read
Share
Today we're starting a new series called Debug 101, which deals with those issues that gave us particularly bad headaches and took a large amount of time to debug, understand and fix. We believe strongly in open source software and open issue resolution, and we try to describe our problems and suggest fixes as we go, so you don't have to shave that yak. We already have, and they yak looks awesome.
We use different cloud providers (typically Kubernetes-managed) to develop and launch workloads on Kubernetes. One of the E2E automated tests we run utilizes AWS EC2, wherein we install the Kubernetes cluster (using kubadm). Custom installations give us the flexibility of custom setups and profiles, and allow us to test Pipeline with different k8s versions. From time to time, we encountered strange behavior in the cluster - the foremost of which was that, although we started EC2 instances, they did not show up in the Kubernetes cluster. This issue occured sporadically and was undeterministic. After some test runs we identified a (potential) common source. Either the node
instances in question were of a stronger instance variety, or the cluster size was particularly small (less than three or four nodes). Since we run our automated tests on spot price
instances, and the pool is recommended
by an internal tool/project (which we will open source soon), it's hard for us to know what our instance types will be beforehand (unless explicitely specified), since Hollowtrees determines this according to a variety of factors (rulesets, spot price history, price trend in the AZ, fleet diversity, etc). To cut to the chase, the problem is that, though master
and node
start in parallel, after some time the controller removes the node from the cluster. Here's the sequence of events:
(kubeadm join --token ${TOKEN} ${MASTER})
(kubectl get node)
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
and the Node join is complete:
* Certificate signing request is sent to master and response received.
* Kubelet informed of new secure connection details.
kubectl get nodes
on the Master register the machine as joinedkubectl event
, which allows us see what's happening cronologically1m 1m 1 ip-10-0-100-184.eu-west-1.compute.internal.14fb501a0ae334a3 Node Normal RegisteredNode controllermanager Node ip-10-0-100-184.eu-west-1.compute.internal event: Registered Node ip-10-0-100-184.eu-west-1.compute.internal in Controller
1m 1m 1 ip-10-0-100-184.eu-west-1.compute.internal.14fb501a19555324 Node Normal DeletingNode controllermanager Node ip-10-0-100-184.eu-west-1.compute.internal event: Deleting Node ip-10-0-100-184.eu-west-1.compute.internal because it's not present according to cloud provider
1m 1m 1 ip-10-0-100-184.eu-west-1.compute.internal.14fb501b435fa741 Node Normal RemovingNode controllermanager Node ip-10-0-100-184.eu-west-1.compute.internal event: Removing Node ip-10-0-100-184.eu-west-1.compute.internal from Controller
1m 1m 1 ip-10-0-100-184.eu-west-1.compute.internal.14fb501c241eff24 Node Normal Starting kubelet, ip-10-0-100-18
This can be paired down to Removing Node ip-10-0-100-184.eu-west-1.compute.internal from Controller
. As you can see, once the controller loop calls the Amazon controller (realizes that the cluster is running on EC2), it removes the node immediately. If we dig a bit deeper into the code and annotate it back after introducing our changes, we have this PR. If the PR is operating correctly, the Amazon controller loop removes instances from the cluster if it thinks that the node is gone or no longer available. This is likely done so that the EC2 instance can die, removed as the result of spot price surgery or due to other communication issues which affect RAFT gossip. We've built Hollowtrees with the exact purpose of avoiding these kinds of problems in the cloud (subscribe to our social channels or check back next week for more on this and other topics). With a simple loop
, this issue can actually be fixed.
until kubectl get node | grep $(hostname -f)
do
kubeadm reset
kubeadm join --token ${TOKEN} ${MASTER}
echo "Waiting for Master to start up..."
sleep 10
done
On the first check there is no kubeconfig
configuration, so the initial join
occurs. In the second iteration the credentials do not work from the first join, and it checks if the node has joined the cluster. If the node has not joined, it retries and repeats this process until the actual join happens. A successful join will look like this.
NAME STATUS ROLES AGE VERSION
ip-10-0-0-51.eu-west-1.compute.internal Ready master 4m v1.8.3
ip-10-0-100-184.eu-west-1.compute.internal Ready <none> 3m v1.8.3
Get emerging insights on innovative technology straight to your inbox.
Outshift is leading the way in building an open, interoperable, agent-first, quantum-safe infrastructure for the future of artificial intelligence.
* No email required
The Shift is Outshift’s exclusive newsletter.
The latest news and updates on generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.