November 8

Dual-stack Kubernetes with kubeadm-dind-cluster

V1.0

Overview

A coworker has pushed out a Kubernetes Enhancement Proposal (KEP) for dual-stack Kubernetes that is currently under review by the community. This capability is currently targeted for the 1.14 release.https://github.com/kubernetes/kubernetes/pull/70659

This proposal will provide IPv4 and IPv6 addresses for all containers (pod network) and nodes (management network), allowing communication with other pods and external resources with either protocol. To simplify this first release will use a single IP family for services, meaning the service network will either be IPv4 or IPv6.

What’s Up?

We’ve started implementing some changes to support dual-stack (as WIP, in some cases, because the KEP is not approved yet). To support that, I’ve modified the kubeadm-dind-cluster provisioning tool (a.k.a k-d-c) so that we can experiment with bringing up a cluster with dual-stack networking, during development.

The changes include setting the CNI configuration files for dual-stack, adding static routes for the Bridge or PTP plugin so that pods can communicate with either IP family across nodes, adjust the KubeAdm configuration file so that the API will use a specific IP family, and does not make use of the DNS64/NAT64 capabilities as both IP families are available on each container.

I’ve verified that we can bring up a cluster in dual-stack mode, with pod to pod (across nodes) and pod to external connectivity using both IPv4 and IPv6. I’ve used IPv4 for the service network, and with PR 70659 (under review as of today), I have verified a cluster with an IPv6 service network.

Granted, there are things that don’t work yet, as much of the KEP needs to be implemented (like service endpoints and pod status API), but it was very satisfying to see a PoC cluster come up.

How To…

To try this out, there are a few preparation steps. First, clone the kubeadm-dind-cluster repo.

cd
git clone https://github.com/kubernetes-sigs/kubeadm-dind-cluster.git dind

 

Next, clone Kubernetes in a subdirectory underneath k-d-c:

cd ~/dind
git clone https://github.com/kubernetes/kubernetes.git

Within the Kubernetes repo, grab my PR that is out for review (or wait until this is merged):

cd kubernetes
git fetch origin pull/70659/head:pr70659
git checkout pr70659

Now, you can bring up a cluster in dual-stack mode, using the desired service network IP family. You can set the dual stack mode:

export IP_MODE=dual-stack

And since we are customizing the Kubernetes code, we need to tell k-d-c to build a new image:

export DIND_IMAGE=mirantis/kubeadm-dind-cluster:local
export BUILD_KUBEADM=y
export BUILD_HYPERKUBE=y

If you are in a lab, and need to use a company DNS server, you can also set REMOTE_DNS64_V4SERVER.

Now, let’s build a new k-d-c image:

cd ..
build/build-local.sh
cd kubernetes

 

To use an IPv6 service network, you can just bring up the cluster using the default values:

../dind-cluster.sh up

 

To use IPv4, you’ll need to first set SERVICE_CIDR to an IPv4 CIDR, before bringing up the cluster. You can use the same value that k-d-c uses for IPv4 only networks, like:

export SERVICE_CIDR="10.96.0.0/12"

 

Then, just use the same “up” command to bring things up.

In each of these modes, you’ll see either IPv4 or IPv6 addresses, when doing a “kubectl get pods –all-namespaces -o wide” command. The pods will still have both IPv4 and IPv6 addresses, and from the pods, you’ll be able to ping and ping6 to external IPv4 and IPv6 sites, respectively.

 

Futures…

I haven’t played with external access to the cluster, and obviously there is work to do for the APIs and kube-proxy, along with changes to kubeadm (see the KEP for details).

I’m working on updating my Lazyjack tool that helps with provisioning Kubernetes on bare-metal nodes, so that it too can bring up dual-stack clusters. This will provide feature parity with k-d-c, only using separate physical nodes, instead of Kubernetes running on node containers (using Docker-in-docker) on a single host.

 

Category: Kubernetes | Comments Off on Dual-stack Kubernetes with kubeadm-dind-cluster
October 22

Lazyjack 1.3.2 New Features

Several new capabilities have been added to Lazyjack recently:

  • Supports IPv4 only mode, so clusters can be created with IPv4 addresses.
  • The kubeadm.conf file generated will use templates that are version specific. This allows easier customizing of the configuration easily. Supports Kuberenetes 1.10-1.13, although experiencing some issues using the alpha 1.13 setup.
  • Clusters can be configured for insecure mode, where init is not needed, and the config YAML file doesn’t have to be copied over to minions (as it is not updated with a token). This makes it easier to start up a cluster, by just running the prepare and up steps.

Time permitting, I hope to add dual stack capabilities.

 

Category: bare-metal, Kubernetes | Comments Off on Lazyjack 1.3.2 New Features
July 20

How to recover (I think) from a botched Kubernetes update

v1.0

What Happened?

I was using KubeAdm v1.10 and wanted to give the latest Kubernetes from master a try. I (unfortunately) just updated the binaries for kubeadm, kubectl, and kubelet.  I restarted the kubelet daemon (“sudo sytemctl restart kubelet”), and then ran “kubeadm init” hoping to sit back and watch the new cluster come up.

First, I found that the config file needed a newer API version, so I changed that to use “kubeadm.k8s.io/v1alpha2”, instead of “kubeadm.k8s.io/v1alpha1”, and tried “kubeadm init” again.

Well, it failed to come up, and it looking at the issue, I found that the kubelet configuration file, /etc/systemd/system/kubelet.service.d/10-kubeadm.conf was referring to a kubelet config file that did not exist:

Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

 

I had no clue how to create this file, nor why it wasn’t there.

 

What Should Have Been Done?

It looks like, going from v1.10 to a newer version, the upgrade procedures should be used. One needs to go from one minor release to minor release at at time (1.10 -> 1.11, 1.11 -> 1.12,…). In this process, one can use “kubeadm config migrate –old-config kubeadm.conf –new-config new-kubeadm.conf, to update the config file. You can then change the API version, and set the Kubernetes version, before using the config file in the update.

I’m not sure what you would do, if you didn’t have a running v1.10 cluster, as this method seems to imply that is needed. Maybe you’d end up in the same state as I was in.

 

What to do, if you didn’t do the upgrade?

From what I can tell, it appears that v1.11+ needs /var/lib/kubelet/config.yaml, which doesn’t exist in v1.10. It will get generated when kubeadm init is invoked, and removed when kubeadm reset is done. But, when I had a previous v1.10 install, it was not getting created during init, and with this file missing, init fails. That file was the key to try to recover from my mess.

On a fresh system install, I brought up Kubernetes v1.11, using KubeAdm and the same config file that I was using on the corrupt system. I took the config.yaml (this one is what I had, YMMV) that was created, and placed it on the system that was corrupted, and then brought up the cluster with “kubeadm init”.

The cluster came up OK with that change. Oddly enough, doing a “kubeadm reset” remokved the file, and it was recreated the next time I did “kubeadm init”. I’m not sure why, when I had v1.10 setup, and then switched to v1.11, the file was not created. In any case, I’m happy it is working now.

I did see another problem though, and I’m not sure if it is related. Once I brought up the master node, set the bridge CNI plugin config file, and untainted the node, I created some alpine pods. From the pod, I could not ping other pods on the node. Looking at the iptables rules, I was seeing this rule…

-P FORWARD DROP

instead of…

-P FORWARD ACCEPT

I flushed all the iptables rules, and then brought up the cluster with KubeAdm again, and after untainting and creating pods, everything seemed to work just peachy.

 

 

Category: Kubernetes | Comments Off on How to recover (I think) from a botched Kubernetes update
July 20

Using kube-router with kubeadm-dind-cluster

v1.0

What is kube-router?

One of the options for networking in Kubernetes, is to use kube-router. This plugin uses the Bridge CNI plugin and go BGP to provide networking for the cluster.

Why use it?

With kube-router, it uses the IPVS (IP Virtual Server) kernel module, instead of iptables rules. This gives much better performance (hash vs serial lookups) and scales much better. Kube-router also uses goBGP, to provide full mesh connectivity using iBGP, instead of requiring static routes, when using the bridge plugin.

What is kubeadm-dind-cluster?

The kubeadm-dind-cluster tool that I’ve mentioned here before, allows you to create a Kubernetes cluster on single host (VM or bare-metal), by using Docker-in-Docker (it creates docker containers, which will be nodes where KubeAdm is invoked to bring up a cluster).

This tool is nice, because it saves you from doing all the tedious steps of setting up a cluster using KubeAdm manually. There are instructions in the kubeadm-dind-cluster repo, on how to use the tool to bring up a cluster.  The tools supports the bridge, calico, flannel, and weave CNI plugins.

Peanut Butter and Chocolate…

I have a PR out in kubernetes-sig/kubeadm-dind-cluster repo to add support for using kube-router, instead of kube-proxy. To use this, you can perform the following steps (assuming you have a kubeadm-dind-cluster repo pulled):

  1. Patch in the PR changes
    1. git fetch origin pull/159/head:pr159
    2. git log –abbrev-commit pr159 –oneline -n 1 | cut -f 1 -d” “
    3. git cherry-pick <# from log output>
  2. build/build-local.sh
  3. export DIND_IMAGE=mirantis/kubeadm-dind-cluster:local
  4. export CNI_PLUGIN=kube-router
  5. Bring up the cluster “./dind-cluster.sh up”

This will skip the normal bridge CNI plugin setup and creation of static routes, run a YAML file to configure the bridge CNI plugin and startup kube-router pods on each node (which will start up BGP), and will then remove the kube-proxy daemonset.

Once the cluster is up, you can “kubectl exec” into one of the kube-router pods to see the BGP and IPVS setup.

After the PR (159) is upstreamed, you’ll only need to set the CNI_PLUGIN to kube-router, and then bring up the cluster.

 

Limitations

Currently, this only works with IPv4. Although ipset and goBGP support IPv6, kube-router is not set up to run in IPv6 mode. There is a PR to add IPv6 support.

Category: Kubernetes | Comments Off on Using kube-router with kubeadm-dind-cluster
June 6

IPv6 Kubernetes – Improving External Access Performance

v1.2 – June 14th 2018

Summary

With current Kubernetes IPv6 only clusters (v1.9.0+), a brute force approach was taken, to deal with the outside world. Since there are some external sites that are IPv4 only, Kubernetes was set up with a NAT64 and DNS64 server to treat all external destinations as IPv4 only.

Here, we’ll talk about ways to more intelligently handle external sites, using IPv6 access, when possible. The result is an improvement in performance, both in space and time.

 

What We Have Today

Let’s use an example of a pod on a minion node of a three node, bare-metal, IPv6 only Kubernetes cluster, trying to ping google.com.

First, the pod requests a lookup of the destination name, to obtain the IP address. Since not all destinations support IPv6 (e.g. github.com), the DNS64 server in the cluster is configured to always use the A record (IPv4) and ignore any AAAA record (IPv6). The IPv4 address will be embedded into a synthesized IPv6 address, using the configured prefix. In this example, the address 216.58.217.78 is combined with the fd00:10:64:ff9b:: prefix to get fd00:10:64:ff9b::d83a:d94e.

The pod (fd00:40::3:0:0:4e7) will then send a ping request, out it’s interface (to fd00:10:64:ff9b::d839:d94e), as shown at (A) in the diagram below.

The ping request will cross the local bridge, br0, and the routing table on the node will direct the packet, over the pod network, to the master node. The packet will be sent (B) from the minion node’s eth1 interface (fd00:20::3) to the master node’s pod network interface (eth1). The route on the master node, will direct the packet to the NAT64 server (a container), over the veth interface.

The NAT64 server (C) creates mapping of source IPv6 address (at this point the minion node’s pod network interface fd00:20::3) to a private IPv4 address (172.18.0.53) from a locally maintained pool. It will extract the destination IPv4 address (216.58.217.78) and send the ping to the master node (D), where iptables employs SNAT to map the private IPv4 address to the node’s IPv4 address (e.g. 10.1.1.2).

Finally, the packet is sent out the main interface (E) to the next hop, which would also do SNAT for this local IPv4 address.

The ping response would follow the reverse route thought the NAT64 server, to the minon node, and finally the pod.

 

Improvements For IPv6 External Sites

We can, however, configure the DNS64 to allow AAAA records to be used, for external destinations that support IPv6 addressing.

In this example,  the DNS lookup would return the AAAA record for google.com (2607:f8b0:4004:801::200e) and the pod shown at (A) would send a ping to that address, as shown in the diagram below.

The ping request would traverse the local bridge, br0, and the routing table on the minion node would direct the packet out the main interface (eth0), and using SNAT, would use the IP of the node as the source address (2001:db8::100), as shown at (B). The packet would be sent to the next hop, where SNAT may occur, if the minion node’s IPv6 address is not public.

The ping response would follow the reverse route, into the minion node, and to the pod.

This avoids sending the packets to the master node’s NAT64 server, where translation and mapping is performed, both a time and space savings (no mapping table needed).

 

Bare Metal Implementation Details

The Lazyjack tool has been modified (in v1.1.0+) to allow the user to specify whether or not destinations that support IPv6 addressing can be directly accessed, without using NAT64.

Under the dns64 section in the config.yaml, there is a new entry titled “allow_aaaa_use”, which if set to “true”, will use the AAAA records from DNS64 and directly access external IPv6 addresses. If omitted, or set to “false”, the existing mechanism of using only the A DNS record and performing NAT64 on all packets for external destinations.

Before using Lazyjack, the nodes of the cluster must be provisioned for IPv6. One each node, this includes:

  • Enabling IPv6 and IPv6 forwarding on main interface.
  • Giving the main interface (with Internet access) an IPv6 address (we used SLAAC).
  • Having a default IPv6 route that sends traffic out the main interface (done via SLAAC).
  • To preserve the default route, set sysctl accept_ra with a value of two. For example:
sudo sysctl net.ipv6.conf.eth0.accept_ra = 2

 

KubeAdm-dind-cluster (DinD) Implementation Details

As of PR 148 merging, the Kubeadm-dind-cluster tool (note the new repo location) for provisioning clusters has been updated to allow the user to enable the ability to use (IPv6) AAAA records for DNS lookups, so that unaltered IPv6 addresses can be used, rather than forcing the use of (IPv4) A records and requiring DNS64 to be used. This new capability can be enabled by setting the environment variable, DIND_ALLOW_AAAA_USE=true.

The k-d-c tool will then use a modified DNS64 configuration, and create the needed ip6tables entries on the host to allow forwarding of packets to the kubeadm-dind-net bridge, and perform SNAT for outgoing packets.

You can check the PR, and once merged, use the latest code on the master branch.

 

Category: bare-metal, Kubernetes | Comments Off on IPv6 Kubernetes – Improving External Access Performance
March 14

KubeAdm with Local Kubernetes Repo for IPv6

V1.4

Goal

In working with my lazyjack tool to create IPv6 based Kubernetes clusters on bare metal systems, I wanted to run the latest code on master (1.10.0-beta.2 at this time) to run E2E tests and possibly tweak things. So, I needed to be able to run KubeAdm with my own repo, instead of using something prebuilt from upstream.

In addition, I wanted to make sure that I could do some customizing with lazyjack.

 

Preparation

As described in my blog post on lazyjack, I have a three node, bare-metal setup, with a second interface connected to a physical switch, for the Kubernetes management/pod network. The lazyjack tool is installed on the nodes and I have a config.yaml with the network topology, including all IP addresses desired. Developement tools (go, git, etc) are installed on the node used as the master node.

I’m using Ubuntu 16.04 on each of my systems.

Next, I pulled down the latest Kubernetes code:

mkdir -p ~/go/src/k8s.io
cd ~/go/src/k8s.io
git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes

 

Now, we are ready to set things up to use this repo, for the Kuberentes cluster.

 

Steps

Repo Prep

I checked out a branch that had the version I wanted. At the time of this writing, beta 2 of 1.10 was available:

cd ~/go/src/k8s.io/kubernetes
git checkout -b release-1.10 origin/release-1.10
git checkout v1.10.0-beta.2

 

Building/Installing

Next, I built everything and installed kubectl, kubeadm, and kubelet binaries, so that we’re using the latest for everything. Restarted kubelet to get the new version running:

make clean
make
make release

cd _output/bin/
sudo cp kubeadm kubectl kubelet /usr/bin/
sudo systemctl daemon-reload
sudo systemctl restart kubelet

 

I copied these three binaries over to the two minion systems, placed them in /usr/bin, and restarted kubelet to update them as well.

For the release images, they are in TAR files, which can be loaded into docker:

cd ~/go/src/k8s.io/kubernetes/_output/release-images/amd64
for f in *.tar; do docker load -i $f; done

 

Startup Local Registry

The docker daemon needs to know about an insecure registry that is being created on one of the nodes. Create the following file on each system:

cat > /etc/docker/daemon.json <<EOT
{
    "insecure-registries": ["10.86.7.77:5000"]
}
EOT
systemctl restart docker

 

I guess I could have used the management IP for the master node ([fd00:20::2]), but I decided to use the admin interface IP of the machine on the lab network. In this case, it was 10.86.7.77. You would replace this, with your master node’s IP address.

On the master node, you want to start the local registry:

docker run -d -p 5000:5000 --restart always --name registry registry:2

 

Tagging images

Note: If there is an easier way than this (like some make target), please let me know…

After making the release and doing a docker load for each tar file, there are a bunch of images in the local registry. We need to tag these images with our master node’s IP and port 5000 (10.86.7.77:5000 in my case). This is a bit complicated, as most of the images built do not have the -amd64 suffix and the tag used has an underscore, which doesn’t play well in the sandbox. Hopefully there is an easier way, but this is what I did…

I first found out the list of images, by doing “docker images” to find out the tag that was created (e.g.v1.10.0-beta.2.17_3d19fe4010c246-dirty). Here’s the list of images:

k8s.gcr.io/hyperkube-amd64
gcr.io/google_containers/kube-apiserver
k8s.gcr.io/kube-apiserver
gcr.io/google_containers/kube-controller-manager
k8s.gcr.io/kube-controller-manager
gcr.io/google_containers/cloud-controller-manager
k8s.gcr.io/cloud-controller-manager
k8s.gcr.io/kube-aggregator
gcr.io/google_containers/kube-aggregator
gcr.io/google_containers/kube-scheduler
k8s.gcr.io/kube-scheduler
gcr.io/google_containers/kube-proxy
k8s.gcr.io/kube-proxy

With that, I could filter by the tag and build up the commands needed to tag these images for my local repo (10.86.7.77:5000). You can see that I had to strip out the registry name to just have the image name part for the local repo tag:

docker images \
    --format="docker tag {{.Repository}}:{{.Tag}} 10.86.7.77:5000/{{.Repository}}:v1.10.0-beta.2" | \
    grep v1.10.0-beta.2.17_3d19fe4010c246 | \
    sed -e "s?5000/gcr.io/google_containers?5000?" | \
    sed -e "s?5000/k8s.gcr.io?5000?" > x
chmod 777 x

 

You can do the same as above, only replace the tag (e.g.v1.10.0-beta.2.17_3d19fe4010c246-dirty) with what you have for a tag. Also, since we need images with the -amd64 suffix, the same command can be tweaked to create those tags too:

docker images \
    --format="docker tag {{.Repository}}:{{.Tag}} 10.86.7.77:5000/{{.Repository}}-amd64:v1.10.0-beta.2" | \
    grep v1.10.0-beta.2.17_3d19fe4010c246 | \
    sed -e "s?5000/gcr.io/google_containers?5000?" | \
    sed -e "s?5000/k8s.gcr.io?5000?" > y
chmod 777 y

 

Before invoking “y”, you need to pull the “hyperkube-amd64” line (first one?), as it already has the right suffix. Now, you can invoke these two files and create all the tags needed. Now, I don’t know if you need the tags in file “x” (other than the hyperkube-amd64), so you could try skipping running that file and just do the first line, along with running file “y”. In my Kubernetes cluster, I think all the images had the -amd64 suffix.

 

Pushing Images

With everything tagged, you can then push these images to your local repo. I did this command, to first check that I have the right syntax for my tags:

docker images | grep 10.86.7.77:5000
10.86.7.77:5000/hyperkube-amd64 v1.10.0-beta.2 351430a5275d 3 days ago 633 MB
10.86.7.77:5000/kube-apiserver v1.10.0-beta.2 2e8a0bd89199 3 days ago 224 MB
10.86.7.77:5000/kube-apiserver-amd64 v1.10.0-beta.2 2e8a0bd89199 3 days ago 224 MB
10.86.7.77:5000/kube-controller-manager v1.10.0-beta.2 80ea4fb85ccb 3 days ago 147 MB
10.86.7.77:5000/kube-controller-manager-amd64 v1.10.0-beta.2 80ea4fb85ccb 3 days ago 1
...

 

If that looks good (registry, image name, and tag), create and invoke the following file in order to push them up:

docker images --format="docker push {{.Repository}}:{{.Tag}}" | \
    grep 10.86.7.77:5000 > z
chmod 777 z
./z

 

Don’t Forget These Images…

Once I tried this all out, I hit a problem with missing images. It turns out that KubeAdm is using some older versions of etcd (3.2.16) and kube-dns (1.14.8) images. To be sure I had these in my local registry (because everything will come from there), I needed to handle these specially:

docker pull gcr.io/google_containers/etcd-amd64:3.2.16
docker tag gcr.io/google_containers/etcd-amd64:3.2.16 10.86.7.77:5000/etcd-amd64:3.2.16
docker push 10.86.7.77:5000/etcd-amd64:3.2.16
docker tag gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8 10.86.7.77:5000/k8s-dns-kube-dns-amd64:1.14.8
docker push 10.86.7.77:5000/k8s-dns-kube-dns-amd64:1.14.8
docker tag gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.8 10.86.7.77:5000/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker push 10.86.7.77:5000/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.8 10.86.7.77:5000/k8s-dns-sidecar-amd64:1.14.8
docker push 10.86.7.77:5000/k8s-dns-sidecar-amd64:1.14.8

As you can see, I didn’t have the etcd image, and had to pull it first. You can check with “docker images” to see if you have to pull any images, before tagging and pushing. Note: For some systems, I’ve had to pull from k8s.gcr.io instead of gcr.io/google_containers.

The versions of etcd and kube-dns that KubeAdm uses are hard coded in the code, so I had to find out by trial and error. Looking at logs, I noticed that these pods were not coming up, and saw what image versions were being tried in the pulls (e.g. 10.86.7.77:5000/etcd-amd64:3.2.16). I would then, do a pull of that version from k8s.gcr.io or gcr.io/google_containers and push it up to my local repo.

 

Bringing Up The Cluster

To bring things up quickly, I’m using a current release (v1.0.6) of my lazyjack tool, but you can do the same manually, if you’re a masochist :). I installed it in /usr/local/bin so that it is in my path.

I created a config.yaml for my setup to represent the topology and addresses that I wanted to use (see lazyjack README.md for details on this file).

For this effort, we need to customize the generate kubeadm.conf file. As a convenience, I overrode the default work area to point to are area under the account I was using (though you can use the default /tmp/lazyjack/ area, if desired):

general:
   plugin: bridge
   work-area: "/home/c2/bare-metal/work-area"

 

I ran “sudo lazyjack init” to create the certificates needed and place them into the config.yaml file. I copied this updated YAML file to the minion nodes, which also have lazyjack installed.

Next, I ran “sudo lazyjack prepare” on the master, and each of the minion nodes. There should be bind9 and tayga containers running on the master, for the DNS64 and NAT64 servers, respectively. This step will also create a kubeadm.conf file in the work area.

To bring up the cluster with the local repo, we need to edit that file to change the kubernetesVersion line to the version tag we are using (instead of 1.9.0 that the tool created), and add the imageRepository line pointing to our repo. In this example, I used:

kubernetesVersion: v1.10.0-beta.2
imageRepository: 10.86.7.77:5000

 

Now, on the master, I ran “sudo lazyjack up”. This takes a few minutes, and the output should indicate success and provide the lines to setup kubectl:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

Run kubectl and make sure that all the pods and services are up and running. Then, you can run the “up” command on the other nodes and check with kubectl that the nodes area “ready” and the additional proxy pods are running.

You should be able to then use your IPv6 based cluster, running code from your local repo!

Issues

Please be sure to use docker version 17.03, as versions 17.06, 17.09, 17.12, 18.03, and 18.04 are showing that, even with the host enabling IPv6, any containers created have IPv6 disabled. The effect seen is that the kube-dns pod is stuck in “creating” state, and logs show that the CNI plugin is failing to add an IPv6 address to the pod (permission denied error). This is true with any user create pods that use the pod network and need an IPv6 address from the CNI plugin. This is discussed in this CNI issue and in this docker issue.

 

Category: bare-metal, Kubernetes | Comments Off on KubeAdm with Local Kubernetes Repo for IPv6
March 6

Istio on IPv6 Kubernetes – Undiscovered Country

V1.2

Overview

Since I was able to get a Kubernetes cluster running with IPv6 only on bare metal, the next logical step was to give a go at trying to bring up Istio. Like the Star Trek movie, this was something untried, and my goal in this blog is to document my efforts to try Istio on IPv6 as a Proof of Concept (PoC). Spoiler: I have it working, but the road to make this a reality, will take quite a few code changes.

Update: Had presented a summary of IPv6 readiness at Istio Community Meeting 3-22-18 At 15:55 mark.

Assumptions

This isn’t for the faint at heart, but I’ll try to make it as cookbook as possible (granted, I need to verify this on a fresh setup, in case my memory failed me on some steps). That said, I expect that you have bare metal systems available, network topology set up, needed tools installed (e.g. Go, GIT, docker, lazyjack), accounts set up on github.com and hub.docker.com, and have installed the Kubernetes that you want to use (1.9+).

For my setup, I have three Ubuntu 16.04 machines, each with IPv4 access to the outside, and a separate interface connected to a switch for Kubernetes management and pod networks. Go is at 1.9.2. I have a cloned Kubernetes master branch on February 14th, 2018 (commit f33e0b3), and built all the needed apps. Kubectl, kubeadm, and kubelet are at v1.9.3 and placed in /usr/bin/. It’s not critical to have the latest and greatest here, as long as it is 1.9+ code.

For Istio, I tried to use the path of least resistance and decided to use NodePort, instead of LoadBalancer, and not to use authentication. I plan on trying MetalLB that I previous tried on a IPv4 cluster.

 

Starting point

Since I had my LazyJack tool working, I used that to bring up my cluster with IPv6. It uses the reference bridge plugin, and has static routes so that nodes can communicate with each other.  Here is the config.yaml that I used for my cluster:

plugin: bridge
topology:
 bxb-c2-77:
 interface: "enp10s0"
 opmodes: "master dns64 nat64"
 id: 2
 bxb-c2-78:
 interface: "enp9s0"
 opmodes: "minion"
 id: 3
 bxb-c2-79:
 interface: "enp10s0"
 opmodes: "minion"
 id: 4
support_net:
 cidr: "fd00:10::/64"
 v4cidr: "172.18.0.0/16"
mgmt_net:
 cidr: "fd00:20::/64"
pod_net:
 prefix: "fd00:40:0:0"
 size: 80
service_net:
 cidr: "fd00:30::/110"
nat64:
 v4_cidr: "172.18.0.128/25"
 v4_ip: "172.18.0.200"
 ip: "fd00:10::200"
dns64:
 remote_server: "64.102.6.247"
 cidr: "fd00:10:64:ff9b::/96"
 ip: "fd00:10::100"

 

Everything needed for this setup, was done by LazyJack in about five minutes, and worked just fine.  I have a Kubernetes cluster running IPv6. Now, let the hacking begin!

 

Istio Preparation

Using the Developer’s Guide as reference, and knowing I already had Go installed, I went right to cloning and setting up the Istio repo…

export ISTIO=$GOPATH/src/istio.io
export HUB="docker.io/pmichali"
export TAG=pmichali
export GITHUB_USER=pmichali
export KUBECONFIG=${HOME}/.kube/config

mkdir -p ~/go/src/github.com/istio.io
cd ~/go/src/github.com/istio.io
git clone https://github.com/istio/istio
cd istio

 

You would want to substitute “pmichali” with your github.com and hub.docker.com username (I did same for tag name). Do a “docker login” to your hub.docker.com account, so that pushes will work later on.

I checked out the latest from master and built everything. In this case, I’m using commit 18a20f9 from March 4th, 2018 and used a separate branch.

git checkout -b trial-20180305

 

 

Spam, Spam, Eggs, and Spam…

Here’s were the fun starts. Several changes are needed to support IPv6. It’s not really that many, however, there are a few changes that, to make them permanent, will require some larger effort. In addition, changes are needed to the sample apps, like BookInfo.

As a starting point, I have a  fork of Istio, where I’ve created an ipv6 branch that is based off of the March 4th, 2018 commit on master (18a20f9). You can take the latest commit (6299579) off of the ipv6 branch, or cherry pick the ones you want. I’ll make issues and submit PRs to Istio for the easy changes that I’ve made.

The first patch (commit a5451cd) modifies validation of proxy addresses in Pilot, to accept IPv6 addresses correctly.

The second patch (commit 0502713) changes the hostname to IP resolution in Pilot, to add the needed square brackets to IPv6 addresses (separating the port from IP part).

The third patch (commit e7e5d48) changes the bootstrap code, so that it can parse IPv6 for Pilot discovery, Zipkin, and statsd addresses that are stored in config.

The other bootstrap code patch (commit 21b82c4), changes a JSON template file, which has the side effect of altering the output of the test files. As a result, the patch also includes updated golden files, so that unit tests pass. For this to be upstreamed, we need to be able to test bootstrap with both IPv4 and IPv6, and have a way to allow deployment of the template file in either mode.

For the Envoy Pilot JSON file(commit 65dc3e9), the IP addresses are patched to use IPv6 addresses for localhost and any host. Like the previous patch, IPv6 is forced, and to upstream, this needs to be configurable, so that users can use either IPv4 or IPv6 mode.

There are two constants in Istio, which specify the wildcard and localhost addresses. This was patched (commit 814036f) so that IPv6 addresses are used.  Like the one bootstrap change, this affects the output of the golden files, so they are included in this commit (quite a few of them). Again to make this upstreamable, this should be configurable, so that users could enable IPv6 mode and use IPv6 addresses everywhere.

I forgot to run lint, before each commit, so I did another commit (commit b7302ab) to fix those warnings, although the actual upstream commits would have to fix these warnings, and do a cleaner fix than the quick and dirty changes I did.

That is it for the base Istio code. We’ll talk about the some of the sample applications later in the blog.

 

Build Everything

Now that the code is changed, and we have the minimum unit test modifications so that things will pass, build everything (you’ll need to do “docker login”, before doing the push):

make
make docker
make push

install/updateVersion.sh -a ${HUB},${TAG}

 

I edited install/kubernetes/istio.yaml so that it uses NodePort, instead of LoadBalancer (search and replace), and uncommented the line selecting port (32000).

 

The Moment of Truth

Bring up the Istio components:

kubectl apply -f install/kubernetes/istio.yaml

 

You should see that all the services and pods are up and running, and most importantly, that pods are not restarting or in a crash loop.  This is what I see on my setup:

$ kubectl get pods --all-namespaces -o wide
NAMESPACE      NAME                                READY     STATUS    RESTARTS   AGE       IP                   NODE
istio-system   istio-ca-5dfc8d9499-jlkdf           1/1       Running   0          10m       fd00:40::3:0:0:25f   bxb-c2-78
istio-system   istio-ingress-df5f9b947-rdn4g       1/1       Running   0          10m       fd00:40::4:0:0:12d   bxb-c2-79
istio-system   istio-mixer-7d95868d79-tmgf6        3/3       Running   0          10m       fd00:40::4:0:0:12c   bxb-c2-79
istio-system   istio-pilot-97d94c7f6-nr7nj         2/2       Running   0          10m       fd00:40::3:0:0:25e   bxb-c2-78
kube-system    etcd-bxb-c2-77                      1/1       Running   0          5h        fd00:20::2           bxb-c2-77
kube-system    kube-apiserver-bxb-c2-77            1/1       Running   0          5h        fd00:20::2           bxb-c2-77
kube-system    kube-controller-manager-bxb-c2-77   1/1       Running   0          5h        fd00:20::2           bxb-c2-77
kube-system    kube-dns-dcf744547-nzzr2            3/3       Running   0          5h        fd00:40::2:0:0:2d    bxb-c2-77
kube-system    kube-proxy-5vbjw                    1/1       Running   0          5h        fd00:20::3           bxb-c2-78
kube-system    kube-proxy-kf5cm                    1/1       Running   0          5h        fd00:20::4           bxb-c2-79
kube-system    kube-proxy-s479m                    1/1       Running   0          5h        fd00:20::2           bxb-c2-77
kube-system    kube-scheduler-bxb-c2-77            1/1       Running   0          5h        fd00:20::2           bxb-c2-77
$ kubectl get svc --all-namespaces -o wide
NAMESPACE      NAME            TYPE           CLUSTER-IP        EXTERNAL-IP   PORT(S)                                                            AGE       SELECTOR
default        kubernetes      ClusterIP      fd00:30::1                443/TCP                                                            5h        
istio-system   istio-ingress   LoadBalancer   fd00:30::2:1e82        80:30802/TCP,443:32379/TCP                                         10m       istio=ingress
istio-system   istio-mixer     ClusterIP      fd00:30::1:1fbc           9091/TCP,15004/TCP,9093/TCP,9094/TCP,9102/TCP,9125/UDP,42422/TCP   10m       istio=mixer
istio-system   istio-pilot     ClusterIP      fd00:30::3:ec89           15003/TCP,15005/TCP,15007/TCP,8080/TCP,9093/TCP,443/TCP            10m       istio=pilot
kube-system    kube-dns        ClusterIP      fd00:30::a                53/UDP,53/TCP                                                      5h        k8s-app=kube-dns

 

What About The Apps?

BookInfo

I’d be remiss, if I didn’t spin up the BookInfo app. After monkeying with this for a while, I realized that this app also needed some changes as well.

I did another patch (commit 4ea619d) that changes the bind address to “::” for book info, so it is listening on the the right IP/port. Also, since I was modifying the app, I needed a way to modify the images that were created, to use my changes. I updated the build_push_update_images.sh script to push the images created, to my repo (instead of docker.io/istio).

With this commit, I ran the script and provided a dummy version:

cd ~/go/src/istio.io/istio/samples/bookinfo
./build_push_update_images.sh 0.0.0
cd ../..
kubectl create -f install/kubernetes/istio-sidecar-injector-configmap-debug.yaml
kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml --injectConfigMapName istio-inject)

 

Once everything is up, you can access the BookInfo productpage, by using the service IP or pod network IP and the port (9080). For example:

kubectl get svc --all-namespaces | grep productpage
default productpage ClusterIP fd00:30::2:8091 <none> 9080/TCP

curl [fd00:30::2:8091]:9080

 

To upstream, this app needs to be modified so that the user can select between an IPv4 and IPv6 variant.

Helloworld

This app also needed to be modified to listen on the IPv6 any address, so another patch was committed (commit 6299579).  New images are created:

cd ~/go/src/istio.io/istio/samples/helloworld/src/
./build_service.sh

 

Then, I would tag and push the two images to my docker hub area:

docker tag istio/examples-helloworld-v1:latest docker.io/pmichali/examples-helloworld-v1:pmichali
docker tag istio/examples-helloworld-v2:latest docker.io/pmichali/examples-helloworld-v2:pmichali

docker push pmichali/examples-helloworld-v1
docker push pmichali/examples-helloworld-v2

 

Prior to applying the ~/go/src/istio.io/istio/samples/helloworld/helloworld.yaml, I modified it (in two places) to point to my images (e.g. docker.io/pmichali/examples-helloworld-v1:pmichali anddocker.io/pmichali/examples-helloworld-v2:pmichali) and I changed the imagePullPolicy to Always. The final step, is to then apply this YAML file:

cd ~/go/src/istio.io/istio/samples/helloworld/
kubectl apply -f helloworld.yaml

 

With this app, there is a nodeport, so you can access it from the service or pod network IP and port 5000, or the node IP using the nodeport:

kubectl get svc | grep helloworld
helloworld    NodePort    fd00:30::76ae             5000:30780/TCP   5m

kubectl get pods --all-namespaces -o wide | grep helloworld
default        helloworld-v1-6759b98975-c6vft      1/1       Running   0          4m        fd00:40::4:0:0:131   bxb-c2-79
default        helloworld-v2-7c6c464dc-g2pcl       1/1       Running   0          4m        fd00:40::3:0:0:263   bxb-c2-78

$ curl [fd00:30::76ae]:5000/hello
Hello version: v1, instance: helloworld-v1-6759b98975-c6vft
$ curl [fd00:40::3:0:0:263]:5000/hello
Hello version: v2, instance: helloworld-v2-7c6c464dc-g2pcl
$ curl [fd00:20::3]:30780/hello
Hello version: v2, instance: helloworld-v2-7c6c464dc-g2pcl

 

This app should be modified so that the IP mode is configurable.

 

Cleanup

For the apps, you can do:

cd ~/go/src/istio.io/istio/samples/helloworld/
kubectl delete -f helloworld.yaml
cd ~/go/src/istio.io/istio/
kubectl delete -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml --injectConfigMapName istio-inject)

 

For Istio, run:

cd ~/go/src/istio.io/istio/
kubectl delete -f install/kubernetes/istio.yaml

 

To bring down Kubernetes, you can use “sudo lazyjack down” on minions and then master mode. Follow this with “sudo layjack clean” to remove everything related to the provisioning for Kubernetes.

 

Final Notes/Observations

I was noticing that, with the BookInfo app, I could “curl” to port 9080, using the service IP and the pod network IP, but I was unable to curl to the app from port 9080 using the node IP address. Also, the service didn’t show a nodeport for BookInfo, and using 32000 did not work either. I didn’t see the NodePort type called out in any of the YAML files. I not sure if there should have been a nodeport defined or if that should work.

With the helloworld app, I could access it from port 5000 using the service and pod IPs, and from port 32677 (shown for the service) using the node’s IP. This worked as expected.

The needed code changes will be easy, in fact, I plan on cleaning up what I have (and adding UTs for the changes). For the JSON and YAML file changes, some form of templating mechanism will be needed to allow operation in either IPv4 or IPv6 mode.

Keep in mind, that if you need to do some iterations on code changes, make sure that the deployment YAML files are set to “Always” pull images, or you need to ensure each node gets the updated version. I would do the following on my nodes (sometimes with the -f option):

docker rmi `docker images --format="{{.ID}} {{.Repository}} {{.Tag}}" | grep pmichali | cut -f 1 -d" "`
docker rmi `docker images --format="{{.ID}} {{.Repository}} {{.Tag}}" | grep istio| cut -f 1 -d" "`
docker rmi `docker images --format="{{.ID}} {{.Repository}} {{.Tag}}" | grep helloworld | cut -f 1 -d" "`
docker rmi `docker images --format="{{.ID}} {{.Repository}} {{.Tag}}" | grep bookinfo | cut -f 1 -d" "`

 

Also, if you run the updateVersion.sh script, you’ll need to make sure that istio.yaml has NodePort set, instead of LoadBalancer, and the port 32000 line uncommented.

Some thought will be needed on how to setup the samples for either IPv4 or IPv6 mode of operation.

Category: bare-metal, Istio, Kubernetes | Comments Off on Istio on IPv6 Kubernetes – Undiscovered Country
February 19

Lazyjack – Provisioning bare-metal for IPv6 Kubernetes

v1.4

I’ve been experimenting with IPv6, Kubernetes, and Istio using Docker-In-Docker. One difficulty I’ve been having is accessing the cluster externally, as the whole cluster is running in docker containers on one VM.

I decided to try to get Kubernetes running on multiple bare-metal nodes. Well, this turned out to be quite challenging, as there are many configuration settings and tweaks needed to make this work.

Not wanting to have to endure that agony, each time I set things up, or spend hours with others’ who want to do the same thing, I decided to write a small Go app to automate this setup. Lazyjack is the culmination of that effort.

You can find details on how to set up and use Lazyjack from the Github repo, but I’ll run through the steps here, using a two system setup I have in a lab.

 

Step 1: Get Everything Needed

Hardware: I already had two Ubuntu 16.04 systems, each with a pair of interfaces, one for SSH access to the box for provisioning, and one connected to an L2 switch, which would be used for the “management” network for Kubernetes. This second interface was new, and didn’t have any configuration on it.

Both boxes have access to the Internet (V4, using NAT in the lab), so that I can access repos and pull down stuff.

Update: If you want to be able to access remote IPv6 sites, without doing NAT64 (and using their IPv4 address), enable IPv6 and forwarding on each node, with an IPv6 address on the main interface. If using SLAAC, ensure system_ra=2 for the main interface, using sysctl.

Software: Being development systems, docker 17.03.2-ce and Go 1.9.2 were installed. I think these systems already had openssl installed. Likewise, Kubernetes was installed (sudo apt-get install kubernetes kubelet kubeadm) on these systems.

Update: You should install CNI v0.7.1+ on the systems, otherwise, there may be issues with IPv6 support (e.g. ip6tables configuration).

Lazyjack: The easiest way is to download the latest release, untar, and place the executable in your system path on each system.  For example, for the first release:

mkdir ~/bare-metal
cd ~/bare-metal
wget https://github.com/pmichali/lazyjack/releases/download/v1.0.0/lazyjack_1.0.0_linux_amd64.tar.gz
tar -xzf lazyjack_1.0.0_linux_amd64.tar.gz
sudo cp lazyjack /usr/local/bin

 

Note: The tar file name may be different, based on the version of lazyjack you use.

Alternately, you can get the repo:

go get github.com/pmichali/lazyjack

build it:

cd ~/go/src/github.com/pmichali/lazyjack
go build cmd/lazyjack.go

 

And then move the executable to your system path on each system. The sample-config.yaml can be used as a template for the configuration.

 

Step 2: Create a Configuration File

I’m lazy, on the system I was going to use as the master node, I just took the sample-config.yaml, and renamed it config.yaml. That file has the following network definitions already set up:

Management network –  fd00:20::/64

Support network – fd00:10::/64

Pod network – fd00:40:0:0:X/80

Service network – fd00:30::/110

DNS64 network –  fd00:64:ff9b::/96

The only thing I needed to do was identify the hostnames I was using, and the interface name for the interface that would be used for the management network. The definitions I used were:

topology:
    bxb-c2-77:
        interface: "enp10s0"
        opmodes: "master dns64 nat64"
        id: 2
    bxb-c2-79:
        interface: "enp10s0"
        opmodes: "minion"
        id: 3
support_net:

 

As you can see, bxb-c2-77 will be the master node, and it will have dns64 and nat64 containers running on it, to support IPv6 on the cluster. The sole minion is bxb-c2-79, but you can clearly more nodes listed here. Likewise, you can use a separate node for the dns64 and nat64 services.

Each node has a unique (and arbitrary), ID from 2-65535 (but why use huge numbers?).

Update: You can configure DNS64 to allow use of IPv6 addresses, so that we can directly access external sites that support IPv6:

dns64:
    allow_ipv6_use: true

 

With that, we are ready to get things rolling…

 

Step 3: Initialize For Kubernetes

On the master (bxb-c2-77 in my case), run lazyjack (I’m assuming it is in your path) with the init command (from the area where the config.yaml file is, so that you don’t have to specify the location):

sudo lazyjack init

 

Yes, you need to run all lazyjack commands as root, because privileged access is needed to various resources. If you don’t run as root, you’ll see a permission denied error.

If you are curious as to what it does, you can add the “-v 4” option, before the “init” argument.

This command will create needed certificates and keys needed for Kubernetes, and will place information into the configuration file (config.yaml), with a .bak preserving the previous version (multiple runs of this command will overwrite that, BTW). Also, the file will be, obviously, owned by root, but the permission changed to 0777, so that you can edit the file, if needed later.

You must copy the configuration file to all other nodes, now that it has the updated information.

 

Step 4: Prepare the Systems

Running lazyjack with the “prepare” command, will get a system ready for running Kubernetes. Run this command on each node.

Note: this command will generate a kubeadm.conf file in the work area (default /tmp/lazyjack) of the master node. If desired, you can customize this file to specify different settings desired for the cluster. For example, you can change the kubernetesVersion line, to pick a different version than 1.9.0 that was generated.

 

Step 5: Cluster Bring-up – Master First

On the master, run lazyjack with the “up” command. This will take a few minutes, as it starts up KubeAdm. Once completed, you can setup kubectl by doing:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

On subsequent runs, I usually do a “rm -rf ~/.kube”, prior to these commands.

Now, you can run “kubectl get nodes -o wide” to see that this node is up, and “kubectl get pods –all-namespaces -o wide”, to see when Kubernetes is fully up. You’ll see something like this:

NAMESPACE   NAME                              READY  STATUS   RESTARTS AGE IP                NODE
kube-system etcd-bxb-c2-77                    1/1    Running  0        2m  fd00:20::2        bxb-c2-77
kube-system kube-apiserver-bxb-c2-77          1/1    Running  0        2m  fd00:20::2        bxb-c2-77
kube-system kube-controller-manager-bxb-c2-77 1/1    Running  0        2m  fd00:20::2        bxb-c2-77
kube-system kube-dns-dcf744547-k56t2          3/3    Running  0        3m  fd00:40::2:0:0:29 bxb-c2-77
kube-system kube-proxy-m9z9m                  1/1    Running  0        3m  fd00:20::2        bxb-c2-77
kube-system kube-scheduler-bxb-c2-77          1/1    Running  0        2m  fd00:20::2        bxb-c2-77

 

You can untaint the master, if you want to be able to create pods on that node.

 

Step 6: Cluster Bring-up – Minions

After you are sure that the master is completely up (all pods and services running), go onto each of the minion nodes, and run the same “up” command. The command should complete quickly, and you can check the status of the node, using the “kubectl get nodes” command on the master. It does take a bit for the minions to become ready. Likewise, you can use the “kubectl get pod” output to see that a proxy is running for each minion.

Note: The reason we don’t do all of the steps on one node, is because lazyjack will setup static routes to other nodes, and the interfaces must be set up on those systems first.

 

Step 7: Enjoy!

That’s it. You can now play with Kubernetes, creating pods that will have IPv6 addresses, and who should be able to ping6 to other pods on other nodes and have external access to the Internet.

 

Step 8: Cleanup

You can run the “down” and then “clean” commands on each minon, and then the master to clean things up.

 

Troubleshooting

Problems Bringing Up a Minion

If the “up” command on a minion fails, you can retry it with “-v 4” to see verbose output. Then, you can manually perform some of the steps that are shown. In one case, I had kubeadm join failing and when running manually, I saw:

c2@bxb-c2-78:~/bare-metal$ sudo kubeadm join --token ...
[preflight] Running pre-flight checks.
 [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
 [ERROR Port-10250]: Port 10250 is in use

 

This occurs when the kubelet service is already running and using that port.  You can stop the service, and then do the “lazyjack up” command or, just run the “down” and then “up” command and that should reload the daemon, and restart the service.

 

 

Category: bare-metal, Go, Istio, Kubernetes, Linux | Comments Off on Lazyjack – Provisioning bare-metal for IPv6 Kubernetes
January 23

Istio, Kubernetes with Load Balancer, on Bare Metal…Oh My!

V1.0 01/23/2018

I found a load balancer that works on bare-metal and decided to do a quick write-up of my findings. This blog assumes that you have a basic understanding on how to bring up Kubernetes and Istio, so I won’t go into the nitty gritty details on those steps.

 

Preparations

The following information indicates the versions used (others may work – this is just what I used), and the basic infrastructure.

For hardware, I used two Cisco UCS blades as the hosts for my cluster, with one acting as master and one acting as a minion. On each system, the following was installed/setup…

  • Ubuntu 16.04 64 bit server OS.
  • Go version 1.9.2.
  • KubeAdm, kubelet, and kubectl v1.9.2.
  • Docker version 17.03.2-ce.
  • Account set up on hub.docker.com for docker registry.
  • Using Istio master branch, cloned on January 22nd 2018 (commit 23306b5)
  • Hosts on lab network with access externally.
  • Four available IPs for external IP pool.

 

Step 1: Bring Up KubeAdm

For Kubernetes, I used the reference bridge plugin, which needs a CNI config file and static route on each host. On the minion, I did this:

cat >/etc/cni/net.d/cni2.conf<<EOT
{
    "cniVersion": "0.3.0",
    "name": "dindnet",
    "type": "bridge",
    "bridge": "dind0",
    "isDefaultGateway": true,
    "ipMasq": false,
    "hairpinMode": true,
    "ipam": {
        "type": "host-local",
        "ranges": [
          [
            {
              "subnet": "10.193.0.0/16",
              "gateway": "10.193.0.1"
            }
          ]
        ]
    }
}

sudo ip route add 10.192.0.0/16 via <ip-of-master>

On the master, I did:

cat >/etc/cni/net.d/cni2.conf<<EOT
{
    "cniVersion": "0.3.0",
    "name": "dindnet",
    "type": "bridge",
    "bridge": "dind0",
    "isDefaultGateway": true,
    "ipMasq": false,
    "hairpinMode": true,
    "ipam": {
        "type": "host-local",
        "ranges": [
          [
            {
              "subnet": "10.192.0.0/16",
              "gateway": "10.192.0.1"
            }
          ]
        ]
    }
}
EOT

sudo ip route add 10.193.0.0/16 via <ip-of-minion>

 

On the master, I created this kubeadm.conf file, which has configuration lines for Istio (and specifies the IP of the master for advertised address):

cat >kubeadm.conf<<EOT
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
kubernetesVersion: v1.9.0
api:
 advertiseAddress: "<ip-of-master>"
networking:
 serviceSubnet: "10.96.0.0/12"
tokenTTL: 0s
apiServerExtraArgs:
 insecure-bind-address: "0.0.0.0"
 insecure-port: "8080"
 runtime-config: "admissionregistration.k8s.io/v1alpha1"
 feature-gates: AllAlpha=true
EOT

 

With all the pieces in place, the master node was brought up with:

sudo kubeadm init --config kubeadm.conf

 

Then, the minion was joined by using the command output from the init invocation on the master (using sudo). Back on the master, I did the obligatory commands to access the cluster with kubectl, and made sure everything was up OK:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 2: Start Up Load Balancer

I cloned the repo for MetalLB and then applied the metallb.yaml file, but you can do what the install page shows:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.3.1/manifests/metallb.yaml

 

I decided to use the ARP method, instead of BGP, as the setup is super easy. Using the example they provided, I created this config file:

apiVersion: v1
kind: ConfigMap
metadata:
 namespace: metallb-system
 name: config
data:
 config: |
   address-pools:
   - name: my-ip-space
     protocol: arp
     arp-network: <start-ip-of-subnet>/26
     cidr:
     - <start-ip-of-pool>/30

If my systems were on a /24 subnet, I wouldn’t have needed the arp-network line. Under “cidr” the start address of the pool and the prefix is specified.

I applied the yaml file using kubectl and made sure that the metallb controller and speaker pods were running. You can check the log of the speaker to ensure that things started up OK, and later to see if IPs are being assigned:

kubectl logs -l app=speaker -n metallb-system

 

Step 3: Start Up Istio

I followed the instructions in the Istio Dev Guide page, to build and start up Istio.

The repo was pulled and a branch created based on the latest from the master branch. I  built the code and pushed to my docker repository, ran updateVersion.sh, and then started Istio with:

kubectl apply -f install/kubernetes/istio.yaml
kubectl apply -f install/kubernetes/istio-initializer.yaml

I verified that everything was runing, and that the istio-ingress service was using the LoadBalancer type and had the first IP address from the pool defined for MetalLB as the external IP. The speaker log for metalLB will show that an IP was assigned.

 

Step 4: BookInfo

We would be remiss, if we didn’t start up the book info application and then try to access the product page using the external address:

kubectl apply -f samples/bookinfo/kube/bookinfo.yaml

 

After this is running, I opened my browser window on my laptop, and went to http://<external-ip>/productpage/ to view the app!

 

Ramblings

The MetalLB setup was painless and worked well. I haven’t tried with /31 for the pool, but it does work with /29 and I suspect larger sizes. I also didn’t try using BGP, instead of ARP.

 

 

 

 

 

Category: bare-metal, Istio, Kubernetes | Comments Off on Istio, Kubernetes with Load Balancer, on Bare Metal…Oh My!
August 9

IPv6 Support for Docker In Docker

V1.16 – June 11th 2018

Goals

To be able to use Docker-In-Docker (DinD) to bring up a cluster with IPv6 only networking. This would work in an environment where the cluster is connected to an IPv4 only network. This allows DinD to be used for running IPv6 based end-to-end(E2E) tests with Google Cloud Platform.

These instructions are for early adopters, who want to start playing with IPv6 clusters.

Background

DinD allows you to create a multi-node cluster, using containers for the nodes, instead of VMs. It leverages off of KubeAdm and provides a quick an easy way to setup a cluster for testing and development.

For the team I am on, we wanted to leverage off of this to be able to setup a cluster with support for IPv6 on pods created (and later for the cluster infrastructure). We’re targeting using DinD for E2E testing of IPv6 logic for Kubernetes. This will work with running DinD using local resources or Google Compute Engine. For this blog entry, we’ll use local resources to start things up, and a bare-metal server running Ubuntu as the host.

DinD supports several CNI plugins, but for the purposes of IPv6 clusters, we’ll use the default bridge and host-local v0.7.1+ CNI plugins that have been tested with IPv6 support.

The DinD modifications also include DNS64 and NAT64 containers. These are used so that an IPv6 only cluster can be connected to an IPv4 outside world. Every DNS lookup will go to the DNS64 server, which will use a remote DNS server (customizable) to obtain the IPv4 address for the host (A record). From that, an IPv4-embedded IPv6 address, with a specific prefix, will be generated and used.  When NAT64 sees this prefix, it will handle V6 to V4 address translation to be able to access the external host.

Update: If your topology supports accessing the Internet via IPv6, you can now set the environment variable DIND_ALLOW_AAAA_USE to true, and when a DNS lookup occurs for an external site that supports IPv6, the AAAA record will be used. This allows pods in the cluster to directly access the site, without using an IPv4-embedded IPv6 address and NAT64.

 

Preparations

For this exercise, I’ll describe how to do this using Ubuntu 16.04 on a bare metal system, but have also played with this on both native Mac (doesn’t support IPv6) and inside a Ubuntu VM running under Vagrant/VirtualBox. You can adapt this for other operating systems as needed.

Tools

I’m using Go version 1.10 and docker version 17.03.2-ce (though have used 17.09.0-ce and 17.11.0-ce too). Note: I had used docker.io 1.13.1 and was seeing an error applying iptables rules, early in the process.

You’ll want git, liblz4-tool (“brew install lz4” for Mac), build-essential (for “make”), and sha1sum (should be there on Ubuntu, “brew install md5sha1sum” on Mac). Some of these are needed, if you plan on updating the DinD code.

Obtain the DinD Code

The IPv6 changes are upstreamed now, so you can clone the latest DinD repo:

cd
git clone https://github.com/kubernetes-sigs/kubeadm-dind-cluster dind

 

 

Obtain the Kubernetes Code and Add IPv6 Patches

Since we want to use the latest Kubernetes code, instead of 1.6, 1.7, or 1.8, which are provided with DinD, we need to clone the repo:

cd ~/dind
git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes

 

Since, we’ll want to use this Kubernetes repo with changes, we’ll set environment variables to indicate to build from the local repo:

export BUILD_KUBEADM=y
export BUILD_HYPERKUBE=y

 

 

Additional Setup

On your host, make sure that IPv6 is enabled and IPv6 forwarding. You can do this using:

sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.all.forwarding=1

 

Configuration

For IPv6 only mode, all you need is to set the environment variable IP_MODE to “ipv6”. However, there are several environment variables that you can export to customize things:

Environment Variable Example value Description
DIND_SUBNET fd00:77:: For IPv6 only mode, to specify the subnet (default is fd00:10::).
REMOTE_DNS64_V4SERVER DNS server that will receive forwarded DNS64 requests for external systems (default is 8.8.8.8). Note: I was using a lab system, and needed to use a local DNS, as it was not forwarding requests to an external DNS.
DNS64_PREFIX fd00:77:64:ff9b:: Prefix that will be used for all DNS resolutions by the built-in DNS64 server (default is 64:ff9b::).
SERVICE_CIDR fd00:77:30::/110 Subnet used for service pods (default is fd00:30::/110).

 

See dind-cluster.sh for other environment variables that can be customized, if you have special requirements.

 

Usage

Once the environment variables are set, a cluster can now be brought up, by using the following commands:

export IP_MODE=ipv6
cd ~/dind/kubernetes
../dind-cluster.sh up

 

You can then create pods, access them, and ping across nodes to other pods, and access external sites. The kubectl command can be run from the host, so you don’t have to access the master node.

When invoking other dind-cluster.sh commands (e.g. down, clean, routes), be sure to run them from the Kubernetes area (e.g. ~/dind/kubernetes/).

If you see a problem with the kube-proxy pod not coming up, take a look at “Kube-proxy failures – The Saga of Conntrack Max” section below for how to resolve this.

Note: The default for IP_MODE is “ipv4”, which brings up an IPv4 only cluster.

 

Rebuild Sequence

Say you have modified some of the Kubernetes code and want to recreate the cluster (after you have brought it down and cleaned). Here are the steps I did, from a setup which already has the DinD code and Kubernetes area, but additional changes have been made to Kubernetes code.

First, I’ll set all the environment variables that I happen to use, just because I’m starting in a new window:

cd ~/dind export IP_MODE=ipv6 export BUILD_KUBEADM=y export BUILD_HYPERKUBE=y

In my lab system, I wanted to use different IPs, and need to use a local DNS server, I did these settings as well:

export REMOTE_DNS64_V4SERVER=64.102.6.247
export DNS64_PREFIX=fd00:77:64:ff9b::
export DIND_SUBNET=fd00:77::
export SERVICE_CIDR=fd00:77:30::/110

 

Finally, the cluster can be brought up:

cd kubernetes
../dind-cluster.sh up

 

If another change is needed, you can do “../dind-cluster.sh clean”, update the Kubernetes code, and then run “../dind-cluster.sh up”.

 

Using Google Compute Engine

If you want to use Google Cloud Platform, setup an account (see my recent blog post on this) and then set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the JSON file with credentials.

There is an example script, gce-setup.sh that can be used as an example for how to use DinD with GCE. For IPv6, you can set the following environment variables and then source the script (from the top of the kubernetes repo, since we are using a local repo for the latest Kubernetes code).

export IP_MODE=ipv6
export BUILD_KUBEADM=y
export BUILD_HYPERKUBE=y

# Set additional env variables as mentioned above, if desired.
cd ~/dind/kubernetes
source ../gce-setup.sh

 

If you are a developer, and want to use customized DinD code, you can set the following flag, to tell DinD to build and use the custom (local) DinD image:

export DIND_IMAGE=mirantis/kubeadm-dind-cluster:local

 

Note: GCE currently only has IPv4 access to the outside world, so the DIND_ALLOW_AAAA_USE flag cannot be set, and all accesses to external sites will use NAT64.

 

Restrictions/Limitations

IPv6 and MacOS

Curently, docker on the Mac, does not support IPv6, so you cannot run DinD directly on the Mac in IPv6 mode. You can however, spin up a Ubuntu VM (e.g. VirtualBox) and run DinD inside that VM.

 

Kube-proxy failures – The Saga of Conntrack Max

Kube-proxy, upon startup, checks the conntrack max setting, and if it is more than four times larger than the hashsize, will attempt to increase the conntrack hashsize. For most cases, like GCE or a VM on a Mac, this will not occur. However, if you have a large system with many CPUs, then this adjustment will be attempted.

Unfortunately, there is a docker bug () where, in a nested docker environment (like DinD), the update fails.  Kube-proxy checks to see if the file system is writeable (it says it is), and then tries to update a file and fails with a read-only error. The kube-proxy log will show this message:

write /sys/module/nf_conntrack/parameters/hashsize: operation not supported

 

DinD attempted to work-around this problem, by telling kube-proxy, via command line settings, to skip changing any settings related to conntrack. Unfortunately, kube-proxy was recently changed, such that the config file, if present (it is with Kubernetes 1.9+), will take precedence over the CLI settings. The config file does not have any settings for conntrack, and so defaults are used, which on some systems will cause kube-proxy to attempt to update the hash size (and it fails).

For example, I have a system with 32 CPUs and kube-proxy uses a default conntrack value of 32768 for max-per-core. We end up with a conntrack value of 1048576 (32768 * 32), which is more than four times the system’s hashsize of 65536.

I created issue #50 in the kubeadmin-dind-cluster project to address the problem. This issue has a patch for Kubeadm, which can be used as a temporary workaround on systems that exhibit the kube-proxy crash, when attempting to update hashsize. If you see the kube-proxy crash with docker log messages like below, you can include the patch from the issue into the code:

I1130 18:53:44.150679 1 conntrack.go:52] Setting nf_conntrack_max to 1048576
I1130 18:53:44.152679 1 conntrack.go:83] Setting conntrack hashsize to 262144
error: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported

 

An alternative to doing a patch, is to look at the hashsize value trying to being set (e.g. 262144), and setting that manually on the host, before doing “dind-cluster.sh up”. For example:

sudo su
echo "262144" > /sys/module/nf_conntrack/parameters/hashsize
cat /sys/module/nf_conntrack/parameters/hashsize

 

NAT64 Container

For NAT64, a Tayga container built by Daneyon Hansen is used. It has a private IPv4 pool for mapping local IPv6 addresses to IPv4 addresses that can be NATed, and used for external site access. The pool is set to a /25 (126 hosts) subnet. If a larger pool is needed, the container would need modifications.

 

Lab Systems and DNS

You may see a case where DNS64 is unable to forward the DNS requests to an outside system like 8.8.8.8. In those cases, I’ve had success with using a company DNS server instead. Hence the programmable setting REMOTE_DNS64_V4SERVER. 🙂

 

Warning on noswap

When you bring up/tear down a cluster, you may see this warning about swap:

WARNING: No swap limit support

 

I’ve seen the warning on Ubuntu 16.04, and it was benign, however other people have seen issues with bringing up the cluster under CentOS 7.

To deal with this, you can check what devices are listed in the output of “cat /proc/swaps”. If there is an entry, you can turn off swap, by running the follow command (showing example for /dev/dm-1):

swapoff -v /dev/dm-1
rm /dev/dm-1

 

 

Category: Kubernetes | Comments Off on IPv6 Support for Docker In Docker