November 2018 – Off The Record

Dual-Stack Kubernetes on bare-metal with LazyJack

v1.0

Preliminary support has been added to Lazyjack as of 1.3.5! Now, as of Kubernetes 1.13, the KEP for dual-stack is still under review, and only a few changes have been made to the code, but you can bring up a cluster in dual-stack mode. You will only see one family of IPs for pods displayed via “kubectl get pod”, but if you look on the pods, you will see both IPv4 and IPv6 addresses.

I’ve already updated kubeadm-dind-cluster to support dual-stack for clusters brought up on a single node, using docker-in-docker, but now Lazyjack supports this too, on bare-metal nodes.

The config.yaml file for Lazyjack will have these changes:

A second CIDR can be specified for the management and pod networks, by using the “cidr2” field, under the respective sections. You can specify one family under “cidr” and one under “cidr2”.
The service network CIDR will specify which family is used for the service network. The KEP only supports a single IP family for service networks at this time.
Omit the DNS64 and NAT64 sections, which are not used in dual-stack mode.
The ‘dns64’ and ‘nat64’ operational modes are note specified under the opmodes field any nodes.

Here is an example config that is using IPv6 for the service network:

general:
    mode: dual-stack
    plugin: ptp
    insecure: true
    kubernetes-version: "v1.13.0-alpha.3"
    work-area: "/home/c2/bare-metal/work-area"
topology:
    minion1:
        interface: "enp10s0"
        opmodes: "minion"
        id: 2
    minion2:
        interface: "enp9s0"
        opmodes: "minion"
        id: 3
    my-master:
        interface: "enp10s0"
        opmodes: "master"
        id: 4
mgmt_net:
    cidr: "10.192.0.0/16"
    cidr2: "fd00:20::/64"
pod_net:
    cidr: "10.244.0.0/16"
    cidr2: "fd00:40::/72"
service_net:
    cidr: "fd00:30::/110"

Category: bare-metal, Kubernetes | Comments Off

November 8

Dual-stack Kubernetes with kubeadm-dind-cluster

V1.0

Overview

A coworker has pushed out a Kubernetes Enhancement Proposal (KEP) for dual-stack Kubernetes that is currently under review by the community. This capability is currently targeted for the 1.14 release.https://github.com/kubernetes/kubernetes/pull/70659

This proposal will provide IPv4 and IPv6 addresses for all containers (pod network) and nodes (management network), allowing communication with other pods and external resources with either protocol. To simplify this first release will use a single IP family for services, meaning the service network will either be IPv4 or IPv6.

What’s Up?

We’ve started implementing some changes to support dual-stack (as WIP, in some cases, because the KEP is not approved yet). To support that, I’ve modified the kubeadm-dind-cluster provisioning tool (a.k.a k-d-c) so that we can experiment with bringing up a cluster with dual-stack networking, during development.

The changes include setting the CNI configuration files for dual-stack, adding static routes for the Bridge or PTP plugin so that pods can communicate with either IP family across nodes, adjust the KubeAdm configuration file so that the API will use a specific IP family, and does not make use of the DNS64/NAT64 capabilities as both IP families are available on each container.

I’ve verified that we can bring up a cluster in dual-stack mode, with pod to pod (across nodes) and pod to external connectivity using both IPv4 and IPv6. I’ve used IPv4 for the service network, and with PR 70659 (under review as of today), I have verified a cluster with an IPv6 service network.

Granted, there are things that don’t work yet, as much of the KEP needs to be implemented (like service endpoints and pod status API), but it was very satisfying to see a PoC cluster come up.

How To…

To try this out, there are a few preparation steps. First, clone the kubeadm-dind-cluster repo.

cd
git clone https://github.com/kubernetes-sigs/kubeadm-dind-cluster.git dind

Next, clone Kubernetes in a subdirectory underneath k-d-c:

cd ~/dind
git clone https://github.com/kubernetes/kubernetes.git

Within the Kubernetes repo, grab my PR that is out for review (or wait until this is merged):

cd kubernetes
git fetch origin pull/70659/head:pr70659
git checkout pr70659

Now, you can bring up a cluster in dual-stack mode, using the desired service network IP family. You can set the dual stack mode:

export IP_MODE=dual-stack

And since we are customizing the Kubernetes code, we need to tell k-d-c to build a new image:

export DIND_IMAGE=mirantis/kubeadm-dind-cluster:local
export BUILD_KUBEADM=y
export BUILD_HYPERKUBE=y

If you are in a lab, and need to use a company DNS server, you can also set REMOTE_DNS64_V4SERVER.

Now, let’s build a new k-d-c image:

cd ..
build/build-local.sh
cd kubernetes

To use an IPv6 service network, you can just bring up the cluster using the default values:

../dind-cluster.sh up

To use IPv4, you’ll need to first set SERVICE_CIDR to an IPv4 CIDR, before bringing up the cluster. You can use the same value that k-d-c uses for IPv4 only networks, like:

export SERVICE_CIDR="10.96.0.0/12"

Then, just use the same “up” command to bring things up.

In each of these modes, you’ll see either IPv4 or IPv6 addresses, when doing a “kubectl get pods –all-namespaces -o wide” command. The pods will still have both IPv4 and IPv6 addresses, and from the pods, you’ll be able to ping and ping6 to external IPv4 and IPv6 sites, respectively.

Futures…

I haven’t played with external access to the cluster, and obviously there is work to do for the APIs and kube-proxy, along with changes to kubeadm (see the KEP for details).

I’m working on updating my Lazyjack tool that helps with provisioning Kubernetes on bare-metal nodes, so that it too can bring up dual-stack clusters. This will provide feature parity with k-d-c, only using separate physical nodes, instead of Kubernetes running on node containers (using Docker-in-docker) on a single host.

Category: Kubernetes | Comments Off