Updates: IPv6 with KubeAdm and Calico

With some recent code changes (so this applies to using latest on master), I found that I needed to modify a few things…

Bare Metal

Where I have IP6_AUTODETECT_METHOD set to “first-found”, in calico.yaml, the environment variable needs to be renamed to IP6_AUTODETECTION_METHOD.

 

Vagrant/VM

I started encountering a failure when joining the second node in this setup. I found that it was using the IP 10.0.2.15 for the IPv4 BGP address and this is a problem on this setup. It turns out that VirtualBox will create a main interface (enp0s3) with the IP 10.0.2.15 for every VM created. Now, the Vagrantfile creates a second interface, enp0s8, that has a different IP for each node, on the subnet 10.96.0.0/16(?). To make Calico use the second interface, the calico.yaml file needs to have this clause added to the BGP section:

 # Auto-detect the BGP IP address.
 - name: IP
 value: ""
 - name: IP_AUTODETECTION_METHOD
 value: "can-reach=10.96.0.101"
 - name: IP6
 value: "autodetect"
 - name: IP6_AUTODETECTION_METHOD
 value: "first-found"

 

I used the can-reach value, but I think I could have done “interface=enp0s8” as well.

For IPv6, I added an IPv6 address to enp0s8, for each node, using a line like (with different IPs on each node, of course):

ip addr add 2001:2::15/64 dev enp0s8

 

Trying With Changes

After bringing up the cluster, creating the IPv6 pool, and enabling IPv6 on each node (/etc/cni/net.d/10-calico.conf), I created some pods, using this clause in the manifest:

metadata:
 name: my-nginx6
spec:
 replicas: 3
 template:
 metadata:
 labels:
 run: my-nginx6
 annotations:
 "cni.projectcalico.org/ipv6pools": "[\"2001:2::/64\"]"
 spec:
 containers:
 - name: my-nginx6
 image: nginx
 ports:
 - containerPort: 8080

 

They all had IPv6 addresses, but there were two issues. First, the two replicas were both created on node-02. I ended up creating eight replicas, so that there were two on node-01. With bare metal, I see that pods are pretty much distributed evenly on all nodes, but I don’t see that in the VM cases (utilization is higher on the master/worker node). One problem down, one to go…

Second, on each node, I don’t see a route to the other node. Looking at “calicoctl node status” (remember to set ETCD_ENDPOINTS as mentioned in other blogs) I see that BFG connections are not working:

Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+--------------------------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+--------------------------------+
| 10.96.0.102 | node-to-node mesh | start | 15:14:38 | Active Socket: Connection |
| | | | | refused |
+--------------+-------------------+-------+----------+--------------------------------+

IPv6 BGP status
+--------------+-------------------+-------+----------+--------------------------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+--------------------------------+
| 2001:2::16 | node-to-node mesh | start | 15:14:38 | Active Socket: Connection |
| | | | | refused |
+--------------+-------------------+-------+----------+--------------------------------+

 

If I look in the calico-node container, I see that the bird and bird6 processes are not running and there are no config files in /etc/calico/confd/config/ on node-02 (is OK on node-01).

I also found that forwarding was not set for all interfaces on both of the nodes, so I did this as well:

sysctl net.ipv6.conf.all.forwarding=1

 

Talking to Gunjan Patel, we looked at the calico-node docker log and saw:

2017-02-23T17:09:07Z node-02 confd[54]: ERROR 501: All the given peers are not reachable (failed to propose on members [http://10.0.2.15:6666] twice [last erro\
r: Get http://10.0.2.15:6666/v2/keys/calico/v1/ipam/v4?quorum=false&recursive=true&sorted=false: dial tcp 10.0.2.15:6666: connection refused]) [0]

 

Looks like it is trying to use 10.0.2.15 for peering and failing. Gunja referred me to a commit he made up (https://github.com/gunjan5/calico-tutorials/commit/eac67014f0509156278dc9396185e784fa7f1aec?diff=unified).

After updating my calico.yaml with these changes, I see that the BGP peering connection is established, when checking node status. I continued on and created IPv6 pods and verified that could ping across nodes. Yay!

For reference, here is the calico.yaml file I’m using (today :)) – working.calico.yaml

That file, adding IPv6 addresses to each node’s enp0s8 interface, and (possibly) enabling forwarding on all IPv6 interfaces, should be enough to do the trick. Then, just add IPv6 pool, enable IPv6 on both nodes, and create pods.

On bare-metal, the calico.yaml specified the interface I wanted to use for the network, and I needed to enable forwarding on the one node (not sure how to persist that). I could then ping from node to container and container to container, across nodes.