Skip to main content

Connecting non-Kubernetes nodes to Calico overlay network

Kubernetes networking has some basic rules.  In short, every pod has to communicate to every other. Selecting the right network plugin for the cluster is a critical key component when planning and architecting a new cluster. Luckily there are great presentations and blog posts around the topic of Kubernetes cluster networking on the internet, but the available sources are very limited about how to connect external resources that aren’t part of the cluster into the mesh. It all depends on what we would like to achieve, so finally we have to glue the solutions together.

In this post I would like to tell our story @IBM about converting an existing node to become a full member of our Kubernetes + Calico network.
First of all we had to specify the main goals:
  • Make node full member of the overlay network
  • External node needs a pod IP to be able to reach it like any regular pod in the system
  • The pod IP must be listenable for services on the external node
  • Service discovery is mandatory for both direction, so pods has to resolve external node's hostname to pod IP and the node has to reach Kubernetes services as well
  • On node restart the same pod IP must reuse for the node
As i mentioned we are using Calico overlay network for many reason witch I don't want to cover now. If you are interested in you can find more about available network plugins here and here. Let's jump into the implementation details.

Make node full member of the overlay network 

This part was the easiest one. On case of Calico network in Kubernetes there is a Deployment (calico-kube-controllers) and a DaemonSet (calico-node). The only thing to do here is to run a well configured calico-node service (both container or native services are supported) on the external node and all the magic happens behind the scenes.

External node needs a pod IP to be able to reach it like any regular pod in the system

Network layer solution doesn't included into Kubernetes. I can agree with this engineering decision, networking is hard to do and also hard to generalize. Every company/service/whatever has it's own special requirements, one is latency the other throughput sensitive. Others have strict company policies or extra security regulations. And lastly but not least some team wants to connect non Kubernetes workloads to the mesh.

Kubernetes uses network plugins and the plugin's responsible to manage the network itself. There are two kind of them, namely kubenet and CNI. In our case Calico is configured as CNI plugin.
After the node was registered in Calico data store and became a full member of the network we can use the Calico CNI plugin to create a Workload endpoint to allocate pod IP address from the IP pool assigned to the node by calico-node service.

The pod IP must be listenable for services on the external node

The tricky part. By default CNI plugin does support only container execution (what a surprise in Kubernetes world). It means the previously created network interface and IP address lives in a separated namespace and it is hidden for other processes. One option to solve this is to run services in this new network namespace. I suggest this only for network experts and for new installations where you want to make non containerized service available only for Kubernetes pods. The other option is to copy the network interface to the default namespace. This solution is a bit tricky but covers more common use cases and makes interface available for regular services.

Service discovery is mandatory for both direction

Service discovery is one of the key components of the solution, because this is the piece witch is used by application developers. There are two ways of service discovery;
  • Pods needs to reach node by hostname:
    • During the provision the configuration tool creates a headless service in Kubernetes witch points to the node's pod IP
  • External node has to reach services in Kubernetes:
    • Node can reach ClusterIPs of the Kubernetes cluster via network interface created by CNI, so it can communicate with CoreDNS. During the provision the configuration tool sets CoreDNS as name resolver

On node restart the same pod IP must reuse for the node

On case of node restart, our target was to restore the same state of the Calico network. But the node was already registered, the IP pool was associated and the pod IP was allocated at start time. We choose the simplest solution as possible: changed the services to clean up previously created Calico node and Workload enpoint before start.

This is the end of the part one. I hope it helped to get a better understanding how Kubernetes and Calico network works and how to extend Calico network with non Kubernetes workers. If you have any comment please feel free to discuss.

Next time we will try this in the practice. To be continue...


Popular posts from this blog

Kubernetes and Calico development environment as easy as a flick

I became an active member of the Calico community so I had to built my own development environment from zero. It wasn't trivial for many reasons but mainly because I have MacOS on my machine and not all of the features of Calico are available on my main operating system. The setup also makes some sense on Linux hosts, because if node controller runs locally it might make changes on the system, which always has some risk in the playing cards. The other big challenge was that I wanted to start any version of Kubernetes with the ability of do changes in it next to Calico. Exactly I had to prepare two tightly coupled environments. My idea was to create a virtual machine with Linux on it, configure development environments for both projects in the VM and use VSCode 's nice remote development feature for code editing. In this way projects are hosted on the target operating system, I don't risk my system, I don't have to deal with poor file system sync between host and g

How to sign messages in Java and verify in Go

In my company we use Java and Go as our development platforms and of course sometimes those projects has to communicate with each other. In the current blog post i would like to introduce our solution to sign messages on the Java side and verify them in a Go service. First let's talk a bit about the architecture. Our Java application spins up new virtual machines in the cloud, and the base image of the instance contains a small Go service. That service is the main entry point of our configuration management system, and we don't want to allow any operation from untrusted clients to modify nodes. Two way SSL with a signature in the request sounded fair enough to trust in clients. Both components are open source, we don't have any "secret" in the binaries, so we elected RSA asymmetric key pairs to generate and verify signatures. Java has the private key and Go has the public one. Time to go deeper. Java is an old platform (personally i have many years of experie

First impressions of the new Cloud Native programming language Ballerina

Nowadays everything is Cloud Native; everybody talks about CN tools, frameworks, solutions, and so on. On the other hand those tools totally changed the way we design, develop, test and release modern applications. I think the number of issues which we solved with the new concepts is equal with the number of new challenges, so in short we simply shoveled problems from one hole to the other. Many new tools appeared on the market to make developers' life easier by integrating software with the underlying infrastructure watching file changes and building containers automatically generating resource descriptors on the fly allowing debugging in a running container etc. Next to the new tools, new programming languages such as Metaparticle , Pulumi or Ballerina have been born. The last one had my attention because others are extensions on top of some existing languages, while Ballerina is a brand new programming language, designed fro