If you have any experience about setuping a kubernetes cluster before, you must notice that you need to choose one CNI in your kubernetes cluster, and there’re many candidate that you can choose, including the
calico and so on.
Most of the kubernetes users and operators don’t know what is the different between those CNI plgins and the only thing they care is that the CNI can make the network works well.
So, I will introduce the Container Network Interface (CNI) in the following articles.
- First, I will try to introcude the docker and it will focus on how network works in the docker environment. Besides, I will also introduce the
Linux Network Namespace (ns)and use the
Linux Network Namespaceas the example in the follwiing articles.
- Second, We have the basic knowhow about network namespace and we can start to learn what is CNI, why we need the CNI and how CNI works. we also use the simple CNI to demostrate how CNI works with network namespace.
- Third, We have learned what is the CNI before, and we will start to implement our own CNI which is a simple CNI just like the bridge network (the default network of docker). This article will be a tutorial about how to write a CNI in
We all know that the docker is very easy to use and we can setup any server we want in one command
For example, I want to run a busybox, I can use the
docker run busybox to run a busybox container in my environment.
The more complicated example is the we can run a simple nginx server with the
docekr run and we can see the example in the
nginx docker hub repo.
Just type the following command in your docker-ready environment.
You will run a nginx server which listens on its network with port 80 and you can connect to the nginx server with
Now, type the following again.
We will run another nginx server which listens to its network with port 80 and you can connect to it with
There is one question, How does the docker do that? why can we run two nginx server listening to 80 port in the same time?
If you have any experience about writing the
socket programming, you must know that we can’t
bind/listen the same tuple(IP,TCP/UDP,Port) in two processes.
We need to choose difference port for each process and that’s why there’re so many well-known port numbers, such as
22,80,443 and we should avoid to use those ports in our appliction.
The reason why we can do it in the docker is
Linux Network Namespace.
The magic how the docker do that is via the
Linux Network Namespace. In the
linux kernel, each network namespace has its own network configuration, including the network interfaces, routing tables, netfilters and we can learn more about in this website.
So, when we run a docker container, the system will create a new network namespace and put it inside the docekr container.
In our previous example, the system will create two network namepsace when we run two nginx docker container and each container has its own
Now, we will learn why we can use the
http://localhost:8080 to access the nginx container in the follwing tutorials.
Besides, we will operates the network namespace and linux bridge to simulate what docker do when we create a docker container.
In the default behavior, the docker will create a linux bridge
docker0 when you install the docker.io/docker.ce into your system.
and it will handle the network connectivity for every docker container (use the –net=bridge and it is docker default option)
You can use the following command to see the linux bridge after you install the docker package.
We can create our own linux bridge via the
brctl command and you can get it by installing the
Create our own linux bridge and assign a IP address to it.
If you have installed the docker package, you can see there’s a interface
docker0 in the system and it’s IP address is
172.17.0.0/16. If that, you should change your
br0 IP address to other CIDR subnet.
We can use the following figure to show the system view of the system now.
The default ip address of the
172.17.0.0/16 and it can be configured via the docker config.
We won’t discuss what is layer2 bridging here, the only thing we need to know is that docker will use this bridge to forward the packets between hosts and containers.
Now, what will happen when we create a docker container?
First, the docker will create a docker container and also create a network namespace indise that container.
The whole system looks like below figure. there’re a linux bridge (docekr0) and a docker container (nginx).
In our example, we won’t use the docker but network namespace, so we can create a network namepsace here.
Up to now, the container(network namespace) doesn’t have the network connectivity which measn any process inside that contaner can’t setup a network connection with outside.
In order to make the
docker container nginx/netowkr namespace has the network connectivity, we need to connect two
network namespaces togehter first. the linux host and the docekr container.
network namespace is a logical concept in the
linux system, we can use another linux technology
veth to help us.
veth is represent to a
virtual link and it can connect to two different network namespace, each
veth pair is made up by two
virtual network interface
For example, type the following command to create a
In the above example, we create a veth pair and the
virtual network interface of it is
ve_B. you can use the some network utils to see them, such as
The system view loooks like beflow, we have a veth pair now but two sides of the veth pair still in the same
Next, we need to move one side of the
veth pair into the docker container, specifically, is the network namespace.
Just like we say before, the
veth pair is used to connect two network namespace. we can do that via the
Now, the ve_B is moved into the network namespace
ns1 and rename as
eth1, we can execute commands in the networl namespace to list the interface.
and you should see the interface
eth1with any IP configuration.
At last, we need to attach another side of
veth pair into the
linux bridge docker0, just use the
Good, We have setup differentes network namespace and connect it via the
The next thing we need to handle it to assign an IP addess to the
docekr container/network namespace. Just like above, use the
ip netns exec ns1 ifconfig eth1 xxxxxx netmask xxxxx to set the ip address to the interface eth1.
The problem is how do we decide what IP address we use?
Since we use the
linux bridge for layer2 forwarding, we sholud put all the
docker container/network namespace and bridge in the same subnet.
Which means we should choose any IP address from
How to choose the IP address is designed by docker and you.
You should avoid to use the duplicate IP address since it will cause the ARP problem.
After choosing the IP address, set to the interface in the
docker continer/network namespace
After that, you can repeat above example to create more network namespace with different IP address and try to use the command
ping to test the network connectivity in the layer 2 network.
The last one we need to understand is
iptables, and it’s a optional step.
For a docker container, if we want to access the container from outside network, we should use the
-p flag to indicate the port mapping in the
docker run command.
For example, when we use the following command to create a docker container.
It will also insert some rules into the
iptables and those rules will do
- if the destination port number of a packet is
8080, forward it to the container
- modify the destination ip to the ip address of container
- modify the destination port number from
- modify the destination ip to the ip address of container
But if we don’t need to access it from outside? we don’t the iptables rules to do that. that why I mean it’s a optional step.
Accoding to the above example, we know that the docker network is based on the
linux network namespace.
What will happen when we run a
- setup a linux bridge (usually be created when you install docker)
- create a network namespace
- create a veth pair (virutal ether link)
- attach the veth pair to target network namespace.
- find a unique IP address and assign to the taget network namespace.
- setup the iptables rules if you want to access it from outside.
In the next posts, I will talk about what is CNI and why we need CNI and how CNI works.