Using KubeAdm and Calico plugin for IPv6 addresses

In an attempt to bring up a container with an IPv6 address, I’ve hit a method that (is a bit manual, but) works and thought I’d document the process used.

Environment

• Using Virtualbox 5.0.32 on a MacBook Pro
• Vagrant version 1.8.7 (says latest is 1.9.1, but I didn’t update)
• 16 GB RAM (though the two VMs created are 2GB each, so should work for many cases)

I’m assuming that you have Virtualbox and Vagrant installed and have a basic understanding how they both work. If not, Google is your friend… :^)

Preparations

I started with Vagrantfile from Gunjan Patel, who was instrumental in helping me get this working (huge props!):

mkdir -p ~/workspace/k8s
cd !$
git clone https://github.com/gunjan5/calico-tutorials.git
cd calico-tutorials/kubeadm

With this repo (commit a10bfd1d), I made this small change to the end of the Vagrantfile, just because I’m lazy and didn’t want to manually “vagrant scp” the file over to the guest VM:

+ vm_name = "%s-01" % instance_name_prefix
+ config.vm.define vm_name do |host|
+   host.vm.provision "file", source: "calico.yaml", destination: "calico.yaml"
+ end

From this point, I did a “vagrant up” and sat back waiting for it to create, boot, and provision the two nodes.

Bringing Up The Cluster

Once startup is done, I did the following on node-01 (using “vagrant ssh node-01”):

sudo su
kubeadm init --api-advertise-addresses=10.96.0.101

This starts up KubeAdm and uses the eth1 interface, which is 10.96.0.101 on this node, and .102 on the other node. The eth0 interface, which is setup for NAT, is the same on both nodes, so we can’t use that.

The cluster will be created and you’ll get a message with a  unique join line like this:

You can now join any number of machines by running the following on each node:

kubeadm join --token=8d4ac7.bba57d4de9d378ec 10.96.0.101

Note that join line, as you’ll need it for the other node. Next, taint the node so that it can be a worker, and apply the calico.yaml.

kubectl taint nodes --all dedicated-
kubectl apply -f calico.yaml
kubectl get pods --all-namespaces -o wide

Reconfiguring For IPv6

Once all the pods are up (you can check with “kubectl get pods –all-namespaces -o wide”), you want to create the IPv6 pool. First though, the setup the endpoint so that calicoctl can access the etcd database. You can find out the IP address (can be different on each run) and port (should be 6666) for etcd with the command:

kubectl get svc --all-namespaces

NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.96.0.1 <none> 443/TCP 3m
kube-system calico-etcd 10.108.3.145 <nodes> 6666:32379/TCP 54s
kube-system kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 3m

With this information, export the etcd endpoint:

export ETCD_ENDPOINTS=http://10.108.3.145:6666

Now, try “calicoctl get ippools” to see the existing IPv4 pool, which should be 192.168.0.0/16. Create an IPv6 pool (I’ll use 2001:2::/64 for mine), by using this script, adjusting for the IP address you want:

cat > pool.yaml <<EOT
- apiVersion: v1
  kind: ipPool
  metadata:
    cidr: 2001:2::/64
EOT
calicoctl create -f pool.yaml

 

You can run the “get ippools” command again, to verify that the new pool is there:

root@node-01:/home/vagrant# calicoctl get ippools
CIDR
192.168.0.0/16
2001:2::/64

The final step is to change CNI config to use IPv6 instead of IPv4. To do this, edit /etc/cni/net.d/10-calico.conf and add in these two bold lines at the location shown in the file:

    "ipam": {
        "assign_ipv4": "false",
        "assign_ipv6": "true",
        "type": "calico-ipam"
    },

Trying It Out

Now, you can create a container with “kubectl apply -f foo.yaml”, and it’ll have an IPv6 address. Here is an example with Nginx:

cat > nginx.yaml <<EOT
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
EOT
kubectl apply -f nginx.yaml

You can verify that it has the right IP address by using “kubectl get pods –all-namespaces -o wide”.

Using the Second Node

The first thing, is to have the second node join the cluster. SSH into “node-02” and paste in that join command you saved above.

vagrant ssh node-02
sudo su
kubeadm join --token=8d4ac7.bba57d4de9d378ec 10.96.0.101

You can check that the node has been added, by doing “kubectl get nodes” on node-01. At this point, containers created on the second node will be using IPv4 address. To use IPv6 addresses, you need to do the same modification to /etc/cni/net.d/10-calico.conf, as was done on node-01. Once that is done, containers that get created on node-02 will have IPv6 addresses.

What’s Next?

I want to try setting 10-calico.conf and create the IPv6 pool right from the start to see if it will work.