Container Runtime Sandboxes

tech over view

containers:

• doesn't mean its contained
• run on shared kernel, but in kernel group
• breakout of kernel group, get all containers

sandbox?

• a playground
• simluated testing env
• a dev server

we mean a security layer when we say sandbox here.

system calls - like an "API" for talking to the kernel.

kernel space vs user space.

sandbox goes here: app1 <--> sandbox <--> system calls <--> kernel <--> hardware

sandbox are not FREE

• more resources
• not good for heavy syscall
• no direct access to hw

container calls Linux kernel

root@cks-master:/etc/kubernetes/manifests# k run pod --image=nginx
pod/pod created

root@cks-master:/etc/kubernetes/manifests# k exec pod -it -- bash

root@pod:/# uname -r
5.4.0-1051-gcp

root@pod:/# strace uname -r | head -n 10

try out dirty cow exploit.

OCI

open container initiative -- allowing communication across different container runtimes.

early days, k8s heavy coupled with docker

kubelet --> dockershim --> dockerd
kubelet --> dockershim --> containerd
kubelet --> dockershim --> runc

new days, created CRI (container runtime interface) allows kubelet to talk to ANY container runtime.

configure kubelet to use a diff CR kubelet --container-runtime <runtime> but kubelet can only use ONE runtime at any one time, not mix and match.

different cli's

crictl - CRI runtime

root@cks-master:/etc/kubernetes/manifests# crictl pull nginx
Image is up to date for nginx@sha256:a05b0cdd4fc1be3b224ba9662ebdf98fe44c09c0c9215b45f84344c12867002e
root@cks-master:/etc/kubernetes/manifests# crictl pods
POD ID              CREATED             STATE               NAME                                 NAMESPACE           ATTEMPT
247186a6c9853       About an hour ago   Ready               kube-apiserver-cks-master            kube-system         2
a1282f11a2f2c       About an hour ago   NotReady            kube-apiserver-cks-master            kube-system         1
c97ffdc923d6f       3 hours ago         Ready               coredns-558bd4d5db-nvqqp             kube-system         5
e938e33d5d299       3 hours ago         Ready               coredns-558bd4d5db-kf8j9             kube-system         5
eea6bb0addd94       3 hours ago         Ready               weave-net-2d9st                      kube-system         5
2f7bdc6478b86       3 hours ago         Ready               kube-proxy-rzbsd                     kube-system         5
c6fce2a6dfc8d       3 hours ago         Ready               kube-scheduler-cks-master            kube-system         5
42f5e053e1ff2       3 hours ago         Ready               kube-controller-manager-cks-master   kube-system         5
8f2fb530b4336       3 hours ago         Ready               etcd-cks-master                      kube-system         5
f4533c76a11ee       4 days ago          NotReady            coredns-558bd4d5db-kf8j9             kube-system         4
7c36e278ff27a       4 days ago          NotReady            coredns-558bd4d5db-nvqqp             kube-system         4
a84684b5e22d3       4 days ago          NotReady            weave-net-2d9st                      kube-system         4
13fb2f701e765       4 days ago          NotReady            kube-proxy-rzbsd                     kube-system         4
f4a0df336c460       4 days ago          NotReady            etcd-cks-master                      kube-system         4

kata containers

• a container runtime sandbox
• hypervisor/vm based

gVisor

• from Google
• a userspace kernel for containers

looks like this: app1 <--> system calls <--> gVisor <--> LIMITED system calls <--> host kernel <--> hardware

create and run RuntimeClasses

runsc(gvisor)

1. create runtime class
2. create a pod to use this class

apiVersion: node.k8s.io/v1  # RuntimeClass is defined in the node.k8s.io API group
kind: RuntimeClass
metadata:
  name: myclass  # The name the RuntimeClass will be referenced by
  # RuntimeClass is a non-namespaced resource
handler: myconfiguration  # The name of the corresponding CRI configuration

edit

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

create class: k create -f ./rc.yaml

create and edit a pod to use our new runtime class

root@cks-master:~# k run gvisor --image=nginx -oyaml --dry-run=client > gvisor-pod.yaml
root@cks-master:~# vim gvisor-pod.yaml

gvisor-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: gvisor
  name: gvisor
spec:
  runtimeClassName: gvisor
  containers:
  - image: nginx
    name: gvisor
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

create pod

root@cks-master:~# k create -f ./gvisor-pod.yaml 
pod/gvisor created

# its stuck
root@cks-master:~# k get pod
NAME     READY   STATUS              RESTARTS   AGE
gvisor   0/1     ContainerCreating   0          18s
pod      1/1     Running             0          21m

# check event messages with describe
k describe pod gvisor
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               69s                default-scheduler  Successfully assigned default/gvisor to cks-worker
  Warning  FailedCreatePodSandBox  13s (x5 over 68s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "runsc" not supported

runtime is missing. just install it and it will work.

from resources, install gvisor script

# don't do this at home ;)
# IF THIS FAILS then you can try to change the URL= further down in the script from latest to a specific release

bash <(curl -s https://raw.githubusercontent.com/killer-sh/cks-course-environment/master/course-content/microservice-vulnerabilities/container-runtimes/gvisor/install_gvisor.sh)

breakout

gvisor, kata