Multus Service Hands on with bridge CNI
8 min readFeb 3, 2023
A hands on about how to use multus-service to create cluserIP service for cluster internal usage. the setup with community k8s which installed with kubeadm, and runtime is cri-o
Install Multus-Service ( assume you already installed Multus-CNI)
git clone https://github.com/k8snetworkplumbingwg/multus-service.git
cd multus-service
kubectl apply -f deploy-nft.ymlInstall IPAM plugin whereabouts
git clone https://github.com/k8snetworkplumbingwg/whereabouts && cd whereabouts
kubectl apply \
-f doc/crds/daemonset-install.yaml \
-f doc/crds/whereabouts.cni.cncf.io_ippools.yaml \
-f doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yamlCheck the installation result
ubuntu@ip-10-0-1-235:~/multus-service$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d4b75cb6d-h8shc 1/1 Running 5 24h
coredns-6d4b75cb6d-tdsmc 1/1 Running 5 24h
etcd-ip-10-0-1-235 1/1 Running 5 24h
kube-apiserver-ip-10-0-1-235 1/1 Running 5 24h
kube-controller-manager-ip-10-0-1-235 1/1 Running 5 24h
kube-multus-ds-sbwjb 1/1 Running 5 24h
kube-proxy-4r9lx 1/1 Running 5 24h
kube-scheduler-ip-10-0-1-235 1/1 Running 5 24h
multus-proxy-ds-amd64-c7xs7 1/1 Running 5 24h
multus-service-controller-796f9855b9-x2kbk 1/1 Running 5 24h
whereabouts-f79sl 1/1 Running 5 24h
ubuntu@ip-10-0-1-235:~/multus-service$ kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-multus-ds 1 1 1 1 1 <none> 24h
kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 24h
multus-proxy-ds-amd64 1 1 1 1 1 kubernetes.io/arch=amd64 24h
whereabouts 1 1 1 1 1 kubernetes.io/arch=amd64 24hCreate net-attach-def and apply it . here I created a net-attachment use bridge CNI. and exclude some of IP address. cni1 is the bridge interface on linux host
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: br-10-2-128
spec:
config: '{
"cniVersion": "0.3.1",
"name": "crio",
"type": "bridge",
"bridge": "cni1",
"ipMasq": true,
"ipam": {
"type": "whereabouts",
"range": "10.2.128.0/24",
"exclude": [
"10.2.128.1/32",
"10.2.128.11/32",
"10.2.128.12/32",
"10.2.128.254/32"
]
}
}'Check the result
ubuntu@ip-10-0-1-235:~/multus-service-lab$ kubectl get net-attach-def br-10-2-128 -o json | yq -P
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{},"name":"br-10-2-128","namespace":"default"},"spec":{"config":"{ \"cniVersion\": \"0.3.1\", \"name\": \"crio\", \"type\": \"bridge\", \"bridge\": \"cni1\", \"ipMasq\": true, \"ipam\": { \"type\": \"whereabouts\", \"range\": \"10.2.128.0/24\", \"exclude\": [ \"10.2.128.1/32\", \"10.2.128.11/32\", \"10.2.128.12/32\", \"10.2.128.254/32\" ] } }"}}
creationTimestamp: "2023-02-03T02:14:37Z"
generation: 1
name: br-10-2-128
namespace: default
resourceVersion: "37057"
uid: 8bd9d035-ea3b-4caa-870b-03c980ffea95
spec:
config: '{ "cniVersion": "0.3.1", "name": "crio", "type": "bridge", "bridge": "cni1", "ipMasq": true, "ipam": { "type": "whereabouts", "range": "10.2.128.0/24", "exclude": [ "10.2.128.1/32", "10.2.128.11/32", "10.2.128.12/32", "10.2.128.254/32" ] } }Create a POD and attach to this net-attach-def . this POD will be assigned with a secondary interface.
ubuntu@ip-10-0-1-235:~/multus-service-lab$ cat nginx.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: multus-nginx-bridge
annotations:
k8s.v1.cni.cncf.io/networks: '[ { "name": "br-10-2-128" } ]'
spec:
containers:
- name: nginx
image: nginx:latest
securityContext:
capabilities:
add: ["NET_ADMIN","SYS_ADMIN","NET_RAW"]check the nginx pod assigned two interface with IP, one is default eth0, other one is on net-attach-def br-10–2–128 with ip 10.2.128.2
ubuntu@ip-10-0-1-235:~/multus-service-lab$ kubectl describe po/nginx | grep networks-status -A 20
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "crio",
"interface": "eth0",
"ips": [
"10.85.0.121",
"1100:200::79"
],
"mac": "7a:96:09:77:9c:64",
"default": true,
"dns": {}
},{
"name": "default/br-10-2-128",
"interface": "net1",
"ips": [
"10.2.128.2"
],
"mac": "32:35:d2:2c:f2:1e",
"dns": {}
}]
Status: RunningCreate clusterIP service for nginx pod. here also assigned an externalIPs which actually it will not work. the 10.2.128.1 is the interface IP of cni1.
ubuntu@ip-10-0-1-235:~/multus-service-lab$ cat nginx_svc_clust_net1.yaml
kind: Service
apiVersion: v1
metadata:
name: multus-nginx-bridge-svc
labels:
service.kubernetes.io/service-proxy-name: multus-proxy
annotations:
k8s.v1.cni.cncf.io/service-network: br-10-2-128
spec:
selector:
app: multus-nginx-bridge
ports:
- protocol: TCP
port: 80
externalIPs:
- 10.2.128.1check the result. multus-service use endpoint-slices to create iptable in the host.
ubuntu@ip-10-0-1-235:~/multus-service-lab$ kubectl get svc multus-nginx-bridge-svc -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
k8s.v1.cni.cncf.io/service-network: br-10-2-128
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"k8s.v1.cni.cncf.io/service-network":"br-10-2-128"},"labels":{"service.kubernetes.io/service-proxy-name":"multus-proxy"},"name":"multus-nginx-bridge-svc","namespace":"default"},"spec":{"externalIPs":["10.2.128.1"],"ports":[{"port":80,"protocol":"TCP"}],"selector":{"app":"multus-nginx-bridge"}}}
creationTimestamp: "2023-02-03T02:29:09Z"
labels:
service.kubernetes.io/service-proxy-name: multus-proxy
name: multus-nginx-bridge-svc
namespace: default
resourceVersion: "42134"
uid: 88b0665c-9e2e-4851-8572-63cc78bbce58
spec:
clusterIP: 10.102.209.137
clusterIPs:
- 10.102.209.137
externalIPs:
- 10.2.128.1
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: multus-nginx-bridge
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}ubuntu@ip-10-0-1-235:~/multus-service-lab$ kubectl get endpointslice multus-nginx-bridge-svc-multus-nfmtt -o yaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
- 10.2.128.2
conditions:
ready: true
serving: true
terminating: false
nodeName: ip-10-0-1-235
targetRef:
kind: Pod
name: nginx
namespace: default
resourceVersion: "38914"
uid: 84d5c668-ebc7-4ae6-91b0-7e65da06cc81
kind: EndpointSlice
metadata:
annotations:
endpoints.kubernetes.io/last-change-trigger-time: "2023-02-03T02:23:57Z"
creationTimestamp: "2023-02-03T02:29:09Z"
generateName: multus-nginx-bridge-svc-multus-
generation: 2
labels:
endpointslice.kubernetes.io/managed-by: multus-endpointslice-controller.npwg.k8s.io
kubernetes.io/service-name: multus-nginx-bridge-svc
service.kubernetes.io/service-proxy-name: multus-proxy
name: multus-nginx-bridge-svc-multus-nfmtt
namespace: default
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: false
controller: true
kind: Service
name: multus-nginx-bridge-svc
uid: 88b0665c-9e2e-4851-8572-63cc78bbce58
resourceVersion: "38926"
uid: 550ab55d-bad1-4ccb-8d4c-ca1931215ed4
ports:
- name: ""
port: 80
protocol: TCPcheck the IPtables on host. ignor the last two entries . that’s I manually created try to make externalIP work.
ubuntu@ip-10-0-1-235:~/multus-service-lab$ sudo iptables-save -c | grep 10.2.128
[0:0] -A POSTROUTING -s 10.2.128.2/32 -m comment --comment "name: \"crio\" id: \"618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec\"" -j CNI-1e3107779b9976299311d64e
[7:424] -A POSTROUTING -s 10.2.128.3/32 -m comment --comment "name: \"crio\" id: \"7f693b118c212b5e0f975811ff5a0a408a5c98b33dcbf3289ff76a0b5f03c521\"" -j CNI-d0f7d98704eefb07bf0d1b20
[0:0] -A CNI-1e3107779b9976299311d64e -d 10.2.128.0/24 -m comment --comment "name: \"crio\" id: \"618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec\"" -j ACCEPT
[0:0] -A CNI-d0f7d98704eefb07bf0d1b20 -d 10.2.128.0/24 -m comment --comment "name: \"crio\" id: \"7f693b118c212b5e0f975811ff5a0a408a5c98b33dcbf3289ff76a0b5f03c521\"" -j ACCEPT
[6:360] -A KUBE-SERVICES -d 10.2.128.1/32 -p tcp -m tcp --dport 80 -j KUBE-EXT-6GJHHST2FFRLV62Q
[0:0] -A KUBE-SERVICES -d 10.2.128.1/32 -p tcp -m tcp --dport 80 -j KUBE-EXT-6GJHHST2FFRLV62Qnow create another client POD to test whether able to access ClusteIP and whether ClusterIP load balancer to secondary network interface. the client pod also need to attach to br-10–2–128
ubuntu@ip-10-0-1-235:~/multus-service-lab$ cat fedora_pod_net1.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: fedora-net1
annotations:
k8s.v1.cni.cncf.io/networks: '[ { "name": "br-10-2-128" } ]'
spec:
containers:
- name: fedora-net1
image: ghcr.io/redhat-nfvpe/multus-service-demo:fedora-tools
imagePullPolicy: Always
command:
- /sbin/init
securityContext:
privileged: trueCheck the result
ubuntu@ip-10-0-1-235:~/multus-service-lab$ kubectl get svc multus-nginx-bridge-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
multus-nginx-bridge-svc ClusterIP 10.102.209.137 10.2.128.1 80/TCP 113m
ubuntu@ip-10-0-1-235:~/multus-service-lab$ kubectl exec -it po/fedora-net1 -- sh
sh-5.1# curl 10.102.209.137
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>doing tcpdump on nginx pod will able to see the packet is reaching net1 interface.
# ./tcpdump -i net1 'tcp port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net1, link-type EN10MB (Ethernet), capture size 262144 bytes
04:22:02.232067 IP multus-nginx-bridge-svc.default.svc.cluster.local.60270 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [S], seq 2624518953, win 64240, options [mss 1460,sackOK,TS val 487542776 ecr 0,nop,wscale 7], length 0
04:22:02.232101 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.60270: Flags [S.], seq 2607370656, ack 2624518954, win 65160, options [mss 1460,sackOK,TS val 4236411161 ecr 487542776,nop,wscale 7], length 0
04:22:02.232139 IP multus-nginx-bridge-svc.default.svc.cluster.local.60270 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 487542777 ecr 4236411161], length 0
04:22:02.232190 IP multus-nginx-bridge-svc.default.svc.cluster.local.60270 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 487542777 ecr 4236411161], length 78: HTTP: GET / HTTP/1.1
04:22:02.232194 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.60270: Flags [.], ack 79, win 509, options [nop,nop,TS val 4236411161 ecr 487542777], length 0
04:22:02.232312 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.60270: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 4236411161 ecr 487542777], length 238: HTTP: HTTP/1.1 200 OK
04:22:02.232331 IP multus-nginx-bridge-svc.default.svc.cluster.local.60270 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 487542777 ecr 4236411161], length 0
04:22:22.281028 IP multus-nginx-bridge-svc.default.svc.cluster.local.41690 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [S], seq 3385843148, win 64240, options [mss 1460,sackOK,TS val 487562826 ecr 0,nop,wscale 7], length 0
04:22:22.281056 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.41690: Flags [S.], seq 1312520049, ack 3385843149, win 65160, options [mss 1460,sackOK,TS val 4236431210 ecr 487562826,nop,wscale 7], length 0
04:22:22.281076 IP multus-nginx-bridge-svc.default.svc.cluster.local.41690 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 487562826 ecr 4236431210], length 0
04:22:22.281117 IP multus-nginx-bridge-svc.default.svc.cluster.local.41690 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 487562826 ecr 4236431210], length 78: HTTP: GET / HTTP/1.1
04:22:22.281120 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.41690: Flags [.], ack 79, win 509, options [nop,nop,TS val 4236431210 ecr 487562826], length 0
04:22:22.281237 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.41690: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 4236431210 ecr 487562826], length 238: HTTP: HTTP/1.1 200 OK
04:22:22.281252 IP multus-nginx-bridge-svc.default.svc.cluster.local.41690 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 487562826 ecr 4236431210], length 0
04:22:22.282030 IP multus-nginx-bridge-svc.default.svc.cluster.local.41690 > 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 487562827 ecr 4236431210], length 0
04:22:22.282144 IP 10-2-128-2.multus-nginx-bridge-svc.default.svc.cluster.local.80 > multus-nginx-bridge-svc.default.svc.cluster.local.41690: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 4236431211 ecr 487562827], length 0tips: standard nginx pod does not have tool like tcpdump. this tcpdump is copied from host. here is how to copy tcpdump (single binary version).
ubuntu@ip-10-0-1-235:~$ sudo crictl ps | grep nginx
df3bee9adfa02 docker.io/library/nginx@sha256:4c1c50d0ffc614f90b93b07d778028dc765548e823f676fb027f61d281ac380d 2 hours ago Running nginx 0 618c144cb3cc7 nginx
ubuntu@ip-10-0-1-235:~$ sudo crictl inspect df3bee9adfa02 | grep overlay
"path": "/var/lib/containers/storage/overlay/494079450d2e58e32525d38ea340f092094be5dd90ad0bf7b7f32791498e0083/merged"
"source": "/run/containers/storage/overlay-containers/618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec/userdata/shm",
"source": "/run/containers/storage/overlay-containers/618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec/userdata/resolv.conf",
"source": "/run/containers/storage/overlay-containers/618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec/userdata/hostname",
"source": "/run/containers/storage/overlay-containers/618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec/userdata/.containerenv",
"io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/618c144cb3cc73f16343fc76c99c8b38f96285231c7670e15bceea344fedc5ec/userdata/resolv.conf",
"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/494079450d2e58e32525d38ea340f092094be5dd90ad0bf7b7f32791498e0083/merged",
ubuntu@ip-10-0-1-235:~$ sudo cp static_linked_linux_binary/tcpdump /var/lib/containers/storage/overlay/494079450d2e58e32525d38ea340f092094be5dd90ad0bf7b7f32791498e0083/diffnow shell into nginx pod, you will able to use tcpdump.
ubuntu@ip-10-0-1-235:~$ kubectl exec -it po/nginx -- sh
# ls
bin boot dev docker-entrypoint.d docker-entrypoint.sh etc home lib lib64 media mnt opt proc root run sbin srv sys tcpdump tmp usr var
# ./tcpdump -i net1 'tcp port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net1, link-type EN10MB (Ethernet), capture size 262144 bytes