Kubernetes/Calico plugin with IPv6 on bare-metal
Documenting a setup for investigating Kubernetes with IPv6 in a lab environment. This builds off of notes for using KubeAdm for Kubernetes with Calico plugin on a bare-metal system, which is behind a firewall in a lab (https://blog.michali.net/2017/02/14/kubernetes-on-a-…-behind-firewall).
These notes should work for Ubuntu 16.04, in addition to CentOS, which was what was used in that blog.
Preparation
In the prior blog, the no-proxy environment variable was setup and the cluster was initialized using an alternate subnet (10.20.30.x/24). Later, i found that it is easier to use the original subnet and just reduce the size. I used the alternative setup, added to that blog as an update.
When trying to switch to IPv6, you’ll need the calicoctl command. The easiest way is to install the calicoctl binary (as root):
curl -L --silent https://github.com/projectcalico/calico-containers/releases/download/v1.0.0/calicoctl -o /usr/local/bin/calicoctl chmod +x /usr/local/bin/calicoctl
Otherwise, you can install go, pull the sources, build and install calicoctl (see end of blog for details).
Starting Up the Cluster
When the cluster is initialized, you can use:
kubeadm init --api-advertise-addresses=10.87.49.77 --service-cidr=10.96.0.0/24
Before applying the calico.yaml, there are additional changes needed. As mentioned in the other blog, the etcd_endpoints and ippool need to be modified. Beyond that, you need to make sure that you have the CNI code with the fix from commit b8fc5928 (merged 2/16/2017), which fixes issue #273. I did that by changing the CNI image line:
image: quay.io/calico/cni:latest
This fixes a problem where some kernels were not honoring the FlagUp option, when creating the veth interfaces.
From this point on, you can apply calico.yaml, and then follow the steps in https://blog.michali.net/2017/02/11/using-kubeadm-and-calico-plugin-for-ipv6-addresses/ under “Reconfiguring for IPv6” to enable Ipv6 for future pod creation. Remember to use “kubectl get svc –all-namespace” to obtain the IP and port for etcd and set the ETCD_ENDPOINTS environment variable, as the calicoctl command will work without this, but will not be accessing the correct key-store entries.
Notes
In the pod, I see these interfaces:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 6a:76:fc:00:4b:cc brd ff:ff:ff:ff:ff:ff inet6 2001:2::6d47:e62d:8139:d1c0/128 scope global valid_lft forever preferred_lft forever inet6 fe80::6876:fcff:fe00:4bcc/64 scope link valid_lft forever preferred_lft forever
There are these routes:
2001:2::6d47:e62d:8139:d1c0 dev eth0 proto kernel metric 256 fe80::/64 dev eth0 proto kernel metric 256 default via fe80::c8e9:11ff:fe2c:c809 dev eth0 metric 1024
On the host, there is this related IP address:
22: cali1500372f1da@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether ca:e9:11:2c:c8:09 brd ff:ff:ff:ff:ff:ff link-netnsid 3 inet6 fe80::c8e9:11ff:fe2c:c809/64 scope link valid_lft forever preferred_lft forever
With these related routes:
2001:2::6d47:e62d:8139:d1c0 dev cali1500372f1da metric 1024 blackhole 2001:2::6d47:e62d:8139:d1c0/122 dev lo proto bird metric 1024 error -22
I did see one system where I could not ping between pods or pod and host, with IPv6 addresses. What I noticed was that, on that system, the cali# interfaces created, although up, did not have a Link Local Address. The pod, had a route to a LLA, which on another system (that worked), it was for the cali# interface. Need to investigate what is wrong on this system.
Manually Building Calicoctl
IF you want to do this the hard way, you can manually build and install the calicoctl tool. First I installed Go on the system:
curl -O http://storage.googleapis.com/golang/go1.7.4.linux-amd64.tar.gz tar -xvf go1.7.4.linux-amd64.tar.gz sudo mv go /usr/local
In ~/.bashrc add:
export PATH=/usr/local/go/bin:$PATH
To use, set GOPATH to the top of a work area for source and add it to the path in your .bashrc file (and re-source it so that your environment is up to date):
export GOPATH=$HOME/go export PATH=$PATH:$GOPATH/bin
Now, calicoctl can be installed (detailed instructions https://github.com/projectcalico/calicoctl). Here is a summary of the steps:
mkdir -p ~/go/src/github.com/projectcalico git clone https://github.com/projectcalico/calicoctl.git $GOPATH/src/github.com/projectcalico/calicoctl
Install glide:
mkdir $GOPATH/bin curl https://glide.sh/get | sh cd ~/go/src/github.com/projectcalico/calicoctl glide install -strip-vendor make binary cd $GOPATH go build src/github.com/projectcalico/calicoctl/calicoctl/calicoctl.go mv calicoctl bin/ sudo cp bin/calicoctl /usr/local/bin sudo chmod 755 /usr/local/bin/calicoctl