18. October 2022

[dimakuv]

Agenda

(please write your proposed agenda items in comments under this discussion)

• Mikko Ylinen: SGX, Gramine and Kubernetes (overview/lecture)
• Dmitrii: How to make smaller & faster subprocesses (the problem of Python and R helper processes)?

Mikko Ylinen: SGX, Gramine and Kubernetes (overview/lecture)

Several different SGX plugins for Kubernetes:

• Intel started writing an SGX device plugin a bit late (with a focus on Icelake CPUs which provide more EPC memory),
• Microsoft wrote its own SGX device plugin, designed for SGX v1 with little EPC -- this implementation doesn't scale well,
• Same for Alibaba's SGX plugin as for Microsoft's one.

Before Kubernetes: there were Docker containers that needed to be orchestrated. There were Docker Swarm, Mesos, OpenStack, but these competing orchestrators lost in popularity. Kubernetes probably won because of its extensible design.

Many Kubernetes distributions: every CSP has its own "managed Kubernetes service". There is also Redhat Openshift. They are "wrappers" around Kubernetes, and all SGX plugins can be plugged into them seamlessly.

Control plane: Kubernetes master. Has an API server -- interface for developer/operator. Stores everything in the etcd database.

Data plane: Kubernetes Nodes (workers). Each node runs several Pods, and has a Kube-proxy to get requests from users and other nodes. Each pod is a scheduling unit, consisting of 1..n containers.

Kubelet (Kubernetes Node Agent) is responsible to get info from Kubernetes master. It's a control plane of the Kubernetes node.

"Pod Spec" -- pod.yaml configuration file of a Pod.

Pods typically have sidecars. One common kind of sidecars is the "networking"/"proxy" sidecar (container). Another kind of sidecars is the "pre-init" container. In case of SGX, the AESMD daemon is run in a sidecar.

Runtime flow: Kubelet gets a command from API server and performs local tasks using the Container Runtime Interface (CRI), typically the containerd/cri-o application (both do the same, just two different implementations). E.g., CRI pulls images, prepares the environment, then invokes OCI, typically the runc application. runc actually creates containers/processes ("workloads").

These days, OCI is "dockerless", i.e., it doesn't use the Docker engine (docker run ...). Instead, Docker images are OCI-compatible, so OCI tools can just pull Docker images and run them (prepare the environment and jump into the ENTRYPOINT).

Pod Spec (pod.yaml) is translated into config.json by containerd/cri-o. The config.json is used to create containers -- it lists cgroups, envvars, command-line args, capabilities, mounts and devices.

Kubernetes has a Device Plugins API. "Devices" is an overloaded term -- it can be real hardware, but it can also be software abstractions (kinda like Linux kernel modules). Kubernetes core didn't want to have vendor-specific devices, e.g. GPUs. So all devices are added via the Device Plugins API ("add-ons").

SGX device plugin communicates with the Kubelet via gRPC. SGX device plugin identifies itself to Kubelet, then SGX device plugin registers all its resources (like SGX EPC) to the Kubelet, then Kubelet informs the container about these resources, and the Linux kernel in the container can talk to SGX device plugin to provide devices/resources to the container.

SGX device plugin does only two things for each container: exposes /dev/sgx_enclave kernel module and provides a limit on EPC size. The SGX device plugin has three resources: /enclave, /provision, /epc. The last one is used to monitor EPC memory, for scheduling decisions.

Linux kernel currently doesn't have SGX EPC limit in cgroups. There is some early feedback, but it's still a long way to go into upstream. So the /epc resource doesn't really enforce anything currently.

The AESMD socket is shared between the host, the SGX-app pod and the AESMD pod. In the pods, it is volume-mounted as /var/run/aesmd:/var/run/aesmd.

TODOs:

• Mikko can return with a demo of Kubernetes + SGX.
• Mikko can also talk about Confidential Containers (CoCo).