WASM-based MRF (#490)

* add skeleton

* use wasmtime

* disable unused features

* add wit and codegen

* add initial component compilation

* update authors name

* rebuild if wit changed

* add config options and docs

* split into files

* add metadata functions

* stuff

* add test binary, add explainer

* progress

* enable async support

* fmt

* fix

* remove unwrap

* embed mrf into kitsune for incoming activities

* rename dir

* updates

* fmt

* disallow any network access

* up

* add manifest for mrf

* add manifest parser/serializer

* AAAAAAAAAAAAAAAAAAAA

* fix test, better diagnostics

* restrict version field to semver

* make schemars optional

* derive default

* convert `activityTypes` to set

* add `configSchema` field

* move mrf-manifest crate

* add license

* add mrf-tool to tree

* fix comment

* remove cfg flags

* add encode function

* add docs

* use simdutf8 where possible

* add description

* restructure commands

* restructure cli layout

* split up functions

* to_owned for manifest

* up

* rename to mrf, version in source

* load mrf module config and pass to the module

* log that config has been found

* add span instead of log level fields

* ensure span is used properly

* filter by activity types

* fix url encoding for international actors

* stuff

* fix

* add submodules

* add wit deps

* add wit readme

* exclude wit deps

* add logging to example

* add logging facade to wasm modules

* add tracing subscriber to wasm mrf test

* remove wasi-keyvalue

* create own keyvalue def

* add kv example

* add bucket example

* add storage backend support

* finish kv storage impl for fs

* up

* up yarn

* flake.lock: Update

Flake lock file updates:

• Updated input 'devenv':
    'github:cachix/devenv/5a30b9e5ac7c6167e61b1f4193d5130bb9f8defa' (2024-02-13)
  → 'github:cachix/devenv/4eccee9a19ad9be42a7859211b456b281d704313' (2024-03-05)
• Updated input 'flake-utils':
    'github:numtide/flake-utils/1ef2e671c3b0c19053962c07dbda38332dcebf26' (2024-01-15)
  → 'github:numtide/flake-utils/d465f4819400de7c8d874d50b982301f28a84605' (2024-02-28)
• Updated input 'nixpkgs':
    'github:nixos/nixpkgs/a4d4fe8c5002202493e87ec8dbc91335ff55552c' (2024-02-15)
  → 'github:nixos/nixpkgs/b8697e57f10292a6165a20f03d2f42920dfaf973' (2024-03-03)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/4ee92bf124fbc4e157cbce1bc2a35499866989fc' (2024-02-16)
  → 'github:oxalica/rust-overlay/e86c0fb5d3a22a5f30d7f64ecad88643fe26449d' (2024-03-05)

* up fuzz

* add licenses

* get rid of futures-retry-policies

* up

* up

* fix warning

* update ci

* add redis skeleton

* impl redis backend

* introduce module name to backend

* add configurable redis backend

* rename

* progress

* remove from fetcher

Author: aumetra

.github/workflows/audit.yml

@@ -14,7 +14,7 @@ jobs:
   security_audit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: rustsec/audit-check@v1.4.1
         with:
           token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/cargo-machete.yml

@@ -15,6 +15,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Machete
         uses: bnjbvr/cargo-machete@main

.github/workflows/release.yml

@@ -62,23 +62,23 @@ jobs:
         # we specify bash to get pipefail; it guards against the `curl` command
         # failing. otherwise `sh` won't catch that `curl` returned non-0
         shell: bash
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.7.2/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.11.1/cargo-dist-installer.sh | sh"
       # sure would be cool if github gave us proper conditionals...
       # so here's a doubly-nested ternary-via-truthiness to try to provide the best possible
       # functionality based on whether this is a pull_request, and whether it's from a fork.
       # (PRs run on the *source* but secrets are usually on the *target* -- that's *good*
       # but also really annoying to build CI around when it needs secrets to work right.)
       - id: plan
         run: |
-          cargo dist ${{ !github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name) || (github.event.pull_request.head.repo.fork && 'plan' || 'host --steps=check') }} --output-format=json > dist-manifest.json
+          cargo dist ${{ (!github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name)) || 'plan' }} --output-format=json > plan-dist-manifest.json
           echo "cargo dist ran successfully"
-          cat dist-manifest.json
-          echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
+          cat plan-dist-manifest.json
+          echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT"
       - name: "Upload dist-manifest.json"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
-          path: dist-manifest.json
+          name: artifacts-plan-dist-manifest
+          path: plan-dist-manifest.json
 
   # Build and packages all the platform-specific things
   build-local-artifacts:
@@ -113,10 +113,11 @@ jobs:
         run: ${{ matrix.install_dist }}
       # Get the dist-manifest
       - name: Fetch local artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: target/distrib/
+          merge-multiple: true
       - name: Install dependencies
         run: |
           ${{ matrix.packages_install }}
@@ -139,9 +140,9 @@ jobs:
 
           cp dist-manifest.json "$BUILD_MANIFEST_NAME"
       - name: "Upload artifacts"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
+          name: artifacts-build-local-${{ join(matrix.targets, '_') }}
           path: |
             ${{ steps.cargo-dist.outputs.paths }}
             ${{ env.BUILD_MANIFEST_NAME }}
@@ -160,13 +161,15 @@ jobs:
         with:
           submodules: recursive
       - name: Install cargo-dist
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.7.2/cargo-dist-installer.sh | sh"
+        shell: bash
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.11.1/cargo-dist-installer.sh | sh"
       # Get all the local artifacts for the global tasks to use (for e.g. checksums)
       - name: Fetch local artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: target/distrib/
+          merge-multiple: true
       - id: cargo-dist
         shell: bash
         run: |
@@ -180,9 +183,9 @@ jobs:
 
           cp dist-manifest.json "$BUILD_MANIFEST_NAME"
       - name: "Upload artifacts"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
+          name: artifacts-build-global
           path: |
             ${{ steps.cargo-dist.outputs.paths }}
             ${{ env.BUILD_MANIFEST_NAME }}
@@ -204,13 +207,14 @@ jobs:
         with:
           submodules: recursive
       - name: Install cargo-dist
-        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.7.2/cargo-dist-installer.sh | sh"
+        run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.11.1/cargo-dist-installer.sh | sh"
       # Fetch artifacts from scratch-storage
       - name: Fetch artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: target/distrib/
+          merge-multiple: true
       # This is a harmless no-op for Github Releases, hosting for that happens in "announce"
       - id: host
         shell: bash
@@ -220,9 +224,10 @@ jobs:
           cat dist-manifest.json
           echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
       - name: "Upload dist-manifest.json"
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
-          name: artifacts
+          # Overwrite the previous copy
+          name: artifacts-dist-manifest
           path: dist-manifest.json
 
   # Create a Github Release while uploading all files to it
@@ -242,10 +247,11 @@ jobs:
         with:
           submodules: recursive
       - name: "Download Github Artifacts"
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
-          name: artifacts
+          pattern: artifacts-*
           path: artifacts
+          merge-multiple: true
       - name: Cleanup
         run: |
           # Remove the granular manifests

.github/workflows/rust.yml

@@ -17,7 +17,7 @@ jobs:
     name: Clippy
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@v4
       - uses: DeterminateSystems/magic-nix-cache-action@main
       - uses: taiki-e/install-action@cargo-hack
@@ -29,11 +29,11 @@ jobs:
     name: Formatting
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@v4
       - uses: DeterminateSystems/magic-nix-cache-action@main
       - run: nix develop --impure -c cargo fmt --all -- --check
-  
+
   test:
     name: Test
     runs-on: ubuntu-latest
@@ -61,7 +61,7 @@ jobs:
           --health-timeout 5s
           --health-retries 5
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@v4
       - uses: DeterminateSystems/magic-nix-cache-action@main
       - uses: Swatinem/rust-cache@v2

.github/workflows/web.yml

@@ -54,7 +54,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       # Setup
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - uses: dtolnay/rust-toolchain@stable

.gitignore

@@ -18,6 +18,9 @@ target-analyzer
 # Production configuration file
 /config.toml
 
+# MRF directory
+/mrf-modules
+
 # Devenv stuff
 /result
 /.devenv

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "crates/kitsune-wasm-mrf/wit/wasi-logging"]
+	path = crates/kitsune-wasm-mrf/wit/wasi-logging
+	url = https://github.com/WebAssembly/wasi-logging.git