Skip to main content

Container Runtime Sandboxes

tech over view#

containers:

  • doesn't mean its contained
  • run on shared kernel, but in kernel group
  • breakout of kernel group, get all containers

sandbox?

  • a playground
  • simluated testing env
  • a dev server

we mean a security layer when we say sandbox here.

system calls - like an "API" for talking to the kernel.

kernel space vs user space.

sandbox goes here: app1 <--> sandbox <--> system calls <--> kernel <--> hardware

sandbox are not FREE

  • more resources
  • not good for heavy syscall
  • no direct access to hw

container calls Linux kernel#

[email protected]:/etc/kubernetes/manifests# k run pod --image=nginxpod/pod created
[email protected]:/etc/kubernetes/manifests# k exec pod -it -- bash
[email protected]:/# uname -r5.4.0-1051-gcp
[email protected]:/# strace uname -r | head -n 10

try out dirty cow exploit.

OCI#

open container initiative -- allowing communication across different container runtimes.

early days, k8s heavy coupled with docker

kubelet --> dockershim --> dockerdkubelet --> dockershim --> containerdkubelet --> dockershim --> runc

new days, created CRI (container runtime interface) allows kubelet to talk to ANY container runtime.

configure kubelet to use a diff CR kubelet --container-runtime <runtime> but kubelet can only use ONE runtime at any one time, not mix and match.

different cli's#

crictl - CRI runtime

[email protected]:/etc/kubernetes/manifests# crictl pull nginxImage is up to date for [email protected]:a05b0cdd4fc1be3b224ba9662ebdf98fe44c09c0c9215b45f84344c12867002e[email protected]:/etc/kubernetes/manifests# crictl podsPOD ID              CREATED             STATE               NAME                                 NAMESPACE           ATTEMPT247186a6c9853       About an hour ago   Ready               kube-apiserver-cks-master            kube-system         2a1282f11a2f2c       About an hour ago   NotReady            kube-apiserver-cks-master            kube-system         1c97ffdc923d6f       3 hours ago         Ready               coredns-558bd4d5db-nvqqp             kube-system         5e938e33d5d299       3 hours ago         Ready               coredns-558bd4d5db-kf8j9             kube-system         5eea6bb0addd94       3 hours ago         Ready               weave-net-2d9st                      kube-system         52f7bdc6478b86       3 hours ago         Ready               kube-proxy-rzbsd                     kube-system         5c6fce2a6dfc8d       3 hours ago         Ready               kube-scheduler-cks-master            kube-system         542f5e053e1ff2       3 hours ago         Ready               kube-controller-manager-cks-master   kube-system         58f2fb530b4336       3 hours ago         Ready               etcd-cks-master                      kube-system         5f4533c76a11ee       4 days ago          NotReady            coredns-558bd4d5db-kf8j9             kube-system         47c36e278ff27a       4 days ago          NotReady            coredns-558bd4d5db-nvqqp             kube-system         4a84684b5e22d3       4 days ago          NotReady            weave-net-2d9st                      kube-system         413fb2f701e765       4 days ago          NotReady            kube-proxy-rzbsd                     kube-system         4f4a0df336c460       4 days ago          NotReady            etcd-cks-master                      kube-system         4

kata containers#

  • a container runtime sandbox
  • hypervisor/vm based

gVisor#

  • from Google
  • a userspace kernel for containers

looks like this: app1 <--> system calls <--> gVisor <--> LIMITED system calls <--> host kernel <--> hardware

create and run RuntimeClasses#

runsc(gvisor)

  1. create runtime class
  2. create a pod to use this class
apiVersion: node.k8s.io/v1  # RuntimeClass is defined in the node.k8s.io API groupkind: RuntimeClassmetadata:  name: myclass  # The name the RuntimeClass will be referenced by  # RuntimeClass is a non-namespaced resourcehandler: myconfiguration  # The name of the corresponding CRI configuration

edit

apiVersion: node.k8s.io/v1kind: RuntimeClassmetadata:  name: gvisorhandler: runsc

create class: k create -f ./rc.yaml

create and edit a pod to use our new runtime class

[email protected]:~# k run gvisor --image=nginx -oyaml --dry-run=client > gvisor-pod.yaml[email protected]:~# vim gvisor-pod.yaml

gvisor-pod.yaml:

apiVersion: v1kind: Podmetadata:  creationTimestamp: null  labels:    run: gvisor  name: gvisorspec:  runtimeClassName: gvisor  containers:  - image: nginx    name: gvisor    resources: {}  dnsPolicy: ClusterFirst  restartPolicy: Alwaysstatus: {}

create pod

[email protected]:~# k create -f ./gvisor-pod.yaml pod/gvisor created
# its stuck[email protected]:~# k get podNAME     READY   STATUS              RESTARTS   AGEgvisor   0/1     ContainerCreating   0          18spod      1/1     Running             0          21m
# check event messages with describek describe pod gvisorEvents:  Type     Reason                  Age                From               Message  ----     ------                  ----               ----               -------  Normal   Scheduled               69s                default-scheduler  Successfully assigned default/gvisor to cks-worker  Warning  FailedCreatePodSandBox  13s (x5 over 68s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "runsc" not supported

runtime is missing. just install it and it will work.

from resources, install gvisor script

# don't do this at home ;)# IF THIS FAILS then you can try to change the URL= further down in the script from latest to a specific release
bash <(curl -s https://raw.githubusercontent.com/killer-sh/cks-course-environment/master/course-content/microservice-vulnerabilities/container-runtimes/gvisor/install_gvisor.sh)

breakout#

gvisor, kata#