27. June 2022

[dimakuv]

Device testing CI pipelines

Dmitrii asked for a separate machine and a corresponding CI pipeline that has the following:

• Virtualization extensions enabled (for KVM)
• Ubuntu 22.04 with Linux 5.15 (for SGX virtualization)

Such platform is needed to test SGX-enabled VMs that start with a dummy device for Gramine testing; see https://github.com/gramineproject/device-testing-tools. An example of an IOCTL that needs such a CI is the "flexible IOCTL" PR: #671.

Woju asked to postpone setting up such a CI machine + pipeline until August. Borys suggested to test the whole Gramine test suite in this CI pipeline, not just Gramine communication with the dummy device.

Gramine web-site

• The current Gramine web-site (https://www.gramineproject.io/) is static and partially outdated.
• Some people would like the web-site to be dynamic; some would like it to have a different design; some just need up-to-date information.
• This requires a significant shake-up of the web-site.
  • Linux Foundation can help with maintaining the web-site (they have some funds allocated), but they prefer to use the WordPress engine.
  • Michal is opposed to WordPress because it is considered rather insecure.
  • There was no conclusion on what to do with the web-site. Mona will follow up on this.

Sanitization of /etc/ files

• Currently we put files like /etc/resolv.conf in the sgx.allowed_files list for simplicity. Example:

  gramine/CI-Examples/redis/redis-server.manifest.template

  Lines 130 to 142 in f7eae7e

  	sgx.allowed_files = [
  	# Name Service Switch (NSS) files. Glibc reads these files as part of name-
  	# service information gathering. For more info, see 'man nsswitch.conf'.
  	"file:/etc/nsswitch.conf",
  	"file:/etc/ethers",
  	"file:/etc/hosts",
  	"file:/etc/group",
  	"file:/etc/passwd",
  	# getaddrinfo(3) configuration file. Glibc reads this file to correctly find
  	# network addresses. For more info, see 'man gai.conf'.
  	"file:/etc/gai.conf",
  	]

• Having these files under sgx.allowed_files is not secure. They are read by e.g. Glibc which doesn't expect them to be ill-formatted or maliciously modified.
• The consensus in the meeting was: Gramine should read the set of network-related files under /etc/ (when specified in the manifest file) and sanitize/verify them, overriding the host versions.
  • Only a small set of network-related files under /etc/ should be sanitized like this.
  • One example of a file not to be sanitized is /etc/passwd. This file should be in the sgx.trusted_files list.

One user-friendliness issue that wasn't discussed is -- where in the manifest these files (like /etc/resolv.conf) should go. Do they just "appear" to the in-Gramine app? Or they need to be put in one of the lists?

The issue to track this: #689

Bug in Gramine's Meson builds

Borys and Sankar hit a bug when they need to purge the Gramine repo when switching between branches or updating to the latest master. This seems to be a case of some missing dependency in our Meson build files, but the root cause is not identified. Maybe Woju and Borys will take a deeper look.

For now, the workaround is to git-clone the whole Gramine repo from scratch (e.g., have different Gramine dirs for each branch). Michal also recommended git clean -dfx but not sure if it helps.

EDMM patch: rationale and design

Vijay discussed his approach to the Enclave Dynamic Memory Management (EDMM) feature in Gramine. See #683.

Vijay, Michal and Mona also mentioned that there is a discussion in the SGX architecture/driver community about EMODPE implementation (who takes the burden of clearing the TLB?), but nothing conclusive.

Updates on GitHub development

Update on PRs, issues and tasks for a previous week:

• (no time for this during the meeting, but my take on this is below)
• The effort "Reorganizing the core repo" started and is planned to be finished until 1. July:
  • Relevant LibOS PRs: Flatten LibOS/shim/ directory to just LibOS/ #659, Use #pragma once instead of handcrafted header guards #669, Lowercase "LibOS", "Pal" and "Scripts" #672, [LibOS] Rename syscalls from shim_do_x to libos_syscall_x #680, [LibOS] Change prefix from shim_ do libos_ #681, [LibOS] Remove shim_ prefix from syscalls trace log #682
  • Relevant PAL PRs: Lowercase "LibOS", "Pal" and "Scripts" #672, [PAL] Change Dk prefix to Pal #685
• Cleanups and small fixes continue after the "Big Sockets Rewrite":
  • [LibOS] Handle IP_RECVERR socket option as a no-op #678, [LibOS] Allow setting O_NONBLOCK on not connected UNIX sockets #677, [LibOS] Support FIONREAD ioctl on sockets #684, [LibOS] Add support for SO_BROADCAST socket option #686
• Introduce sgx.remote_attestation = "[none|epid|dcap]" is merged.
• [LibOS] Add truncate to zero support is merged.
• New big PRs:
  • [LibOS,Pal] Add flexible device-specific IOCTL support
  • [Pal/Linux-SGX] RFC: Support for Microsoft Azure Attestation -- not that new anymore, but needs reviews.
• Interesting GSC PRs:
  • Support Docker images that have a non-root user was merged

Agenda for the next meeting

Moved to #716.

[vans163]

Dmitrii asked for a separate machine and a corresponding CI pipeline that has the following:

Virtualization extensions enabled (for KVM)
Ubuntu 22.04 with Linux 5.15 (for SGX virtualization)

We can provide this free of charge at https://gpux.ai/, is it okay if the access is nonroot and /dev/sgx* passed through to a podman container? If not we can accommodate. It would be 3rd Gen Xeons (2021+) with ~32/64 cores and 256GB ECC 3200mhz DDR4.

There was no conclusion on what to do with the web-site. Mona will follow up on this.

How do you feel about remix.run + Cloudflare pages? You can see how we did our homepage with automatic pipeline from cloudflare gpuedge/homepage#1. Also remix.run is fully SEO friendly as serverside rendering is done.

[dimakuv]

We can provide this free of charge at https://gpux.ai/, is it okay if the access is nonroot and /dev/sgx* passed through to a podman container?

Thanks, but we need a machine inside our CI cluster. I don't think we can re-use a machine from the outside of this cluster (which is physically located in North Carolina in US).

How do you feel about remix.run + Cloudflare pages?

Tanks for the suggestion. This is a question mainly to @mkow and @monavij I guess.

[vans163]

We can ship you a build (donate)

8375C Xeon (3rd Gen) (32/64cores 1 socket)
MU72-SU0 Motherboard (or a dual socket /w 2x 10gbe RJ45s)
2TB PM9A1 (OEM Samsung 980 PRO)
256GB DDR4 RDIMM ECC 3200 mhz
2000 Watt 240v PSU 95-Gold  (poland has 240v right? unlike US at 120v)
1x RTX3060 GPU for testing IOCTLs

About remix.run we searched a lot for optimal solution, remix.run has support for customizing the metatags of each route (for SEO) as well it loads the website optimally in 1 go and doesn't load unused code paths. The important checkboxes for us were SEO and ease of deployment (we did not want to earn a PhD in webpack+javascript packaging).

[woju]

Thank you for the offer, we'll be grateful for the machine. We'll talk about this in the next maintainers meeting and will answer next Monday evening. (Facility in Poland has 230V-ish and facility at UNC I would have to confirm).

[vans163]

Thank you for the offer, we'll be grateful for the machine. We'll talk about this in the next maintainers meeting and will answer next Monday evening. (Facility in Poland has 230V-ish and facility at UNC I would have to confirm).

NP. 208-250v-ish should be fine, we can ship 120v PSU but itl be 1200-1600Watt (dont recall). This CPU will take no where near 2000w but its what we have (should be around 40-60w idle at the wall [cpufan+mobo+drive+ram] and ~200-250w under max load).

[monavij]

Thanks a lot for your offer to donate a server. On remix.run + Cloudflare - what is the cost ? We will also take a look at this option and discuss in our next community meeting.

[woju]

> We can provide this free of charge at https://gpux.ai/, is it okay if the > access is nonroot and /dev/sgx* passed through to a podman container? Thanks, but we need a machine inside our CI cluster. I don't think we can re-use a machine from the outside of this cluster (which is physically located in North Carolina in US).

I think we can use machines elsewhere, we just need SSH and/or wireguard access through firewall to run CI executor and query metrics. However, we'd need baremetal (or at least VPS with access to SGX), to be free to arrange CI. Non-root container won't work. (Currently we use Jenkins, but we seriously think about migrating to something else).