|
| 1 | +# Goma and Buildbarn |
| 2 | + |
| 3 | +This is an incomplete overview of the steps needed to run `goma` with `Buidlbarn`. |
| 4 | +It is based on the [discussion in the issue] as well as private chats. |
| 5 | +Big thanks to [Jidong Qin](https://github.com/qinjidong) for figuring it all out! 🎉 |
| 6 | + |
| 7 | +[discussion in the issue]: https://github.com/buildbarn/bb-deployments/issues/96 |
| 8 | + |
| 9 | +## Setup |
| 10 | + |
| 11 | +1) Install google's [depot tools] to manage dependencies |
| 12 | + and working with `goma`. |
| 13 | + These are needed on PATH, but contains a lot of scripts. |
| 14 | + So it is best to keep them isolated and only add them to PATH |
| 15 | + in the terminal where you work with `goma`. |
| 16 | + |
| 17 | +2) Install [goma client]. |
| 18 | + We use `cipd` which comes from `depot tools`: |
| 19 | + |
| 20 | + $ cipd install infra/goma/client/linux-amd64 -root ~/goma |
| 21 | + |
| 22 | +3) Checkout the [goma server]. |
| 23 | + |
| 24 | +3) We need something to build, them most common is the `chromium` project. |
| 25 | + With documentation available here: [build chromium] and [additional building info]. |
| 26 | + |
| 27 | +[goma client]: https://chromium.googlesource.com/infra/goma/client/ |
| 28 | +[goma server]: https://chromium.googlesource.com/infra/goma/server/ |
| 29 | + |
| 30 | +## Overview |
| 31 | + |
| 32 | +There are a few parts in this, |
| 33 | +`ninja` is the main buildsystem, which calls `goma`, |
| 34 | +and a compiler wrapper, `gomacc`. |
| 35 | + |
| 36 | +The `goma` client will itself spin up two background tasks: |
| 37 | +a `http_proxy` that connects the client to the server, |
| 38 | +you'll notice that the ports used for the client point to this proxy, |
| 39 | +which in-turn talks to the `goma` server, `rbe proxy`. |
| 40 | + |
| 41 | +The `compiler proxy` too is central in this, |
| 42 | +it includes a detailed web-page for all compile actions, |
| 43 | +and their errors as well as server logs (info, warn, error). |
| 44 | + |
| 45 | + ninja |
| 46 | + |
| 47 | + -> goma client |
| 48 | + -> http_proxy |
| 49 | + -> compiler_proxy |
| 50 | + |
| 51 | + -> goma server (rbe proxy) |
| 52 | + -> Buildbarn |
| 53 | + |
| 54 | +## Patches |
| 55 | + |
| 56 | +`goma` is designed to work with Google infrastructure |
| 57 | +and authentication. |
| 58 | +We do not use either, but there are no feature flags for this behavior, |
| 59 | +so we must patch three components. |
| 60 | + |
| 61 | +### Patch the client |
| 62 | + |
| 63 | +We cut off the authentication code in `goma_auth.py` |
| 64 | +I did it in the client repository, but it is easier to just install with `cipd`. |
| 65 | + |
| 66 | + |
| 67 | + goma $ git diff |
| 68 | + diff --git a/client/goma_auth.py b/client/goma_auth.py |
| 69 | + index 5cc674d..e5425e9 100755 |
| 70 | + --- a/client/goma_auth.py |
| 71 | + +++ b/client/goma_auth.py |
| 72 | + @@ -1,574 +1,5 @@ |
| 73 | + #!/usr/bin/env python3 |
| 74 | + |
| 75 | + -# Copyright 2015 The Goma Authors. All rights reserved. |
| 76 | + -if __name__ == '__main__': |
| 77 | + ... |
| 78 | + - sys.exit(main()) |
| 79 | + + print("Bypassing authentication 'goma_auth.py'.") |
| 80 | + + return 0 |
| 81 | + |
| 82 | + |
| 83 | + |
| 84 | + $ cat ~/goma/bin/goma_auth.py |
| 85 | + #!/usr/bin/env python3 |
| 86 | + |
| 87 | + def main(): |
| 88 | + print("Bypassing authentication 'goma_auth.py'.") |
| 89 | + return 0 |
| 90 | + |
| 91 | +### Patch the remote execution proxy |
| 92 | + |
| 93 | +First, to work with `Buildbarn` we need a simple patch, |
| 94 | +patch the `OSFamily` platform property to lowercase. |
| 95 | +In the `goma` server repository: |
| 96 | + |
| 97 | + commit 8d1ba1eb6aed0b504448f464ae365e9af705788c (HEAD) |
| 98 | + Author: Nils Wireklint <nils@meroton.com> |
| 99 | + Date: Tue Aug 15 11:50:19 2023 +0200 |
| 100 | + |
| 101 | + Fix OSFamily value capitalization |
| 102 | + |
| 103 | + In accordance with the REv2 API the standard value of the OSFamily |
| 104 | + platform property should be lowercase. |
| 105 | + |
| 106 | + See |
| 107 | + https://github.com/bazelbuild/remote-apis/blob/068363a3625e166056c155f6441cfb35ca8dfbf2/build/bazel/remote/execution/v2/platform.md |
| 108 | + |
| 109 | + diff --git a/cmd/remoteexec_proxy/main.go b/cmd/remoteexec_proxy/main.go |
| 110 | + index 4ab92a2..d321344 100644 |
| 111 | + --- a/cmd/remoteexec_proxy/main.go |
| 112 | + +++ b/cmd/remoteexec_proxy/main.go |
| 113 | + @@ -412,7 +412,7 @@ func main() { |
| 114 | + Value: *platformContainerImage, |
| 115 | + }, { |
| 116 | + Name: "OSFamily", |
| 117 | + - Value: "Linux", |
| 118 | + + Value: "linux", |
| 119 | + }, |
| 120 | + }, |
| 121 | + }, |
| 122 | + { |
| 123 | + |
| 124 | +Run the `goma` server (to proxy to RBE): |
| 125 | + |
| 126 | + goma/server $ go run \ |
| 127 | + cmd/remoteexec_proxy/main.go \ |
| 128 | + -port 5050 \ |
| 129 | + -remoteexec-addr localhost:8980 \ |
| 130 | + -remote-instance-name "hardlinking" \ |
| 131 | + -platform-container-image 'docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448' \ |
| 132 | + -insecure-remoteexec |
| 133 | + |
| 134 | +The container image platform property is set on the command line. |
| 135 | +This should say that it is running, accepts you and can talk RBE: |
| 136 | + |
| 137 | + 2023-08-17T11:33:40.842+0200 INFO exec/inventory.go:190 configure platform config: target:{addr:"grpc://127.0.0.1:8980"} build_info:{} remoteexec_platform:{properties:{name:"container-image" value:"docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448"} properties:{name:"OSFamily" value:"linux"} rbe_instance_basename:"hardlinking"} dimensions:"os:linux" |
| 138 | + |
| 139 | +### Patch the goma server |
| 140 | + |
| 141 | +The majority of authentication and access token handling is done in the server, |
| 142 | +so we can patch away all of that. |
| 143 | +Following the instructions from the [bromite guide] |
| 144 | + |
| 145 | +First create a dummy token file: |
| 146 | + |
| 147 | + echo "nomatter" > ~/.debug_goma_auth_file |
| 148 | + |
| 149 | +Then patch the `goma server`: |
| 150 | + |
| 151 | + commit f9365d8432ad8e4cb8832b19dcec1a0301f634f8 (HEAD) |
| 152 | + Author: Nils Wireklint <nils@meroton.com> |
| 153 | + Date: Fri Aug 18 12:25:50 2023 +0200 |
| 154 | + |
| 155 | + apply patch from bromite/discussions/1032 |
| 156 | + |
| 157 | + diff --git a/auth/acl/checker.go b/auth/acl/checker.go |
| 158 | + index f225309..f70edd3 100644 |
| 159 | + --- a/auth/acl/checker.go |
| 160 | + +++ b/auth/acl/checker.go |
| 161 | + @@ -114,6 +114,8 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo |
| 162 | + |
| 163 | + logger := log.FromContext(ctx) |
| 164 | + |
| 165 | + + return "id1", nil, nil |
| 166 | + + |
| 167 | + g, err := c.FindGroup(ctx, tokenInfo) |
| 168 | + if err != nil { |
| 169 | + if ctx.Err() != nil { |
| 170 | + @@ -163,6 +165,7 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo |
| 171 | + func checkGroup(ctx context.Context, tokenInfo *auth.TokenInfo, g *pb.Group, authDB AuthDB) (bool, error) { |
| 172 | + logger := log.FromContext(ctx) |
| 173 | + logger.Debugf("checking group:%s", g.Id) |
| 174 | + + return true, nil |
| 175 | + if g.Audience != "" { |
| 176 | + if tokenInfo.Audience != g.Audience { |
| 177 | + logger.Debugf("audience mismatch: %s != %s", tokenInfo.Audience, g.Audience) |
| 178 | + diff --git a/auth/client.go b/auth/client.go |
| 179 | + index 1e49076..8e5ca09 100644 |
| 180 | + --- a/auth/client.go |
| 181 | + +++ b/auth/client.go |
| 182 | + @@ -153,6 +153,12 @@ func (a *Auth) Check(ctx context.Context, req *http.Request) (*enduser.EndUser, |
| 183 | + defer span.End() |
| 184 | + logger := log.FromContext(ctx) |
| 185 | + |
| 186 | + + fake_token := &oauth2.Token{ |
| 187 | + + AccessToken: "", |
| 188 | + + TokenType: "", |
| 189 | + + } |
| 190 | + + return enduser.New("fake_email", "fake_groupid", fake_token), nil |
| 191 | + + |
| 192 | + authorization := req.Header.Get("Authorization") |
| 193 | + if authorization == "" { |
| 194 | + logger.Warnf("no authorization header") |
| 195 | + diff --git a/remoteexec/exec.go b/remoteexec/exec.go |
| 196 | + index 251b5e9..a5515d8 100644 |
| 197 | + --- a/remoteexec/exec.go |
| 198 | + +++ b/remoteexec/exec.go |
| 199 | + @@ -604,6 +604,7 @@ func (r *request) newInputTree(ctx context.Context) *gomapb.ExecResp { |
| 200 | + } |
| 201 | + |
| 202 | + symAbsOk := r.f.capabilities.GetCacheCapabilities().GetSymlinkAbsolutePathStrategy() == rpb.SymlinkAbsolutePathStrategy_ALLOWED |
| 203 | + + symAbsOk = true |
| 204 | + |
| 205 | + cmdCleanCWD := cleanCWD |
| 206 | + cmdCleanRootDir := cleanRootDir |
| 207 | + : |
| 208 | + commit f9365d8432ad8e4cb8832b19dcec1a0301f634f8 (HEAD) |
| 209 | + Author: Nils Wireklint <nils@meroton.com> |
| 210 | + Date: Fri Aug 18 12:25:50 2023 +0200 |
| 211 | + |
| 212 | + apply patch from bromite/discussions/1032 |
| 213 | + |
| 214 | + diff --git a/auth/acl/checker.go b/auth/acl/checker.go |
| 215 | + index f225309..f70edd3 100644 |
| 216 | + --- a/auth/acl/checker.go |
| 217 | + +++ b/auth/acl/checker.go |
| 218 | + @@ -114,6 +114,8 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo |
| 219 | + |
| 220 | + logger := log.FromContext(ctx) |
| 221 | + |
| 222 | + + return "id1", nil, nil |
| 223 | + + |
| 224 | + g, err := c.FindGroup(ctx, tokenInfo) |
| 225 | + if err != nil { |
| 226 | + if ctx.Err() != nil { |
| 227 | + @@ -163,6 +165,7 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo |
| 228 | + func checkGroup(ctx context.Context, tokenInfo *auth.TokenInfo, g *pb.Group, authDB AuthDB) (bool, error) { |
| 229 | + logger := log.FromContext(ctx) |
| 230 | + logger.Debugf("checking group:%s", g.Id) |
| 231 | + + return true, nil |
| 232 | + if g.Audience != "" { |
| 233 | + if tokenInfo.Audience != g.Audience { |
| 234 | + logger.Debugf("audience mismatch: %s != %s", tokenInfo.Audience, g.Audience) |
| 235 | + diff --git a/auth/client.go b/auth/client.go |
| 236 | + index 1e49076..8e5ca09 100644 |
| 237 | + --- a/auth/client.go |
| 238 | + +++ b/auth/client.go |
| 239 | + @@ -153,6 +153,12 @@ func (a *Auth) Check(ctx context.Context, req *http.Request) (*enduser.EndUser, |
| 240 | + defer span.End() |
| 241 | + logger := log.FromContext(ctx) |
| 242 | + |
| 243 | + + fake_token := &oauth2.Token{ |
| 244 | + + AccessToken: "", |
| 245 | + + TokenType: "", |
| 246 | + + } |
| 247 | + + return enduser.New("fake_email", "fake_groupid", fake_token), nil |
| 248 | + + |
| 249 | + authorization := req.Header.Get("Authorization") |
| 250 | + if authorization == "" { |
| 251 | + logger.Warnf("no authorization header") |
| 252 | + diff --git a/remoteexec/exec.go b/remoteexec/exec.go |
| 253 | + index 251b5e9..a5515d8 100644 |
| 254 | + --- a/remoteexec/exec.go |
| 255 | + +++ b/remoteexec/exec.go |
| 256 | + @@ -604,6 +604,7 @@ func (r *request) newInputTree(ctx context.Context) *gomapb.ExecResp { |
| 257 | + } |
| 258 | + |
| 259 | + symAbsOk := r.f.capabilities.GetCacheCapabilities().GetSymlinkAbsolutePathStrategy() == rpb.SymlinkAbsolutePathStrategy_ALLOWED |
| 260 | + + symAbsOk = true |
| 261 | + |
| 262 | + cmdCleanCWD := cleanCWD |
| 263 | + |
| 264 | + |
| 265 | +And start the `goma server`: |
| 266 | + |
| 267 | + ~/goma/goma_ctl.py start |
| 268 | + |
| 269 | +[bromite guide]: https://github.com/bromite/bromite/discussions/1032 |
| 270 | + |
| 271 | +## Setup Chromium |
| 272 | + |
| 273 | +Download and skip the history: |
| 274 | + |
| 275 | + $ fetch --nohooks --no-history chromium |
| 276 | + $ cd src |
| 277 | + $ ./build/install-build-deps.sh |
| 278 | + $ gclient sync |
| 279 | + |
| 280 | + $ gn args out/Default |
| 281 | + |
| 282 | +## Building Chromium |
| 283 | + |
| 284 | + chromium/src $ export GOMA_SERVER_HOST=localhost |
| 285 | + export GOMA_SERVER_PORT=5050 |
| 286 | + export GOMA_USE_SSL=false |
| 287 | + export GOMA_HERMETIC=error |
| 288 | + export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true |
| 289 | + export GOMA_HTTP_AUTHORIZATION_FILE=~/.debug_goma_auth_file |
| 290 | + export GOMA_USE_LOCAL=false |
| 291 | + export GOMA_FALLBACK=true |
| 292 | + |
| 293 | + # generate ninja files to use goma |
| 294 | + $ gn gen --args="use_goma=true goma_dir=\"~/goma\" " out/Default |
| 295 | + |
| 296 | + $ ~/goma/goma_ctl.py start |
| 297 | + |
| 298 | + # build something |
| 299 | + $ ninja -j16 -C out/Default obj/base/base/base64.o |
| 300 | + |
| 301 | +# Appendix |
| 302 | + |
| 303 | +## Other guides |
| 304 | + |
| 305 | +There are a few other guides that can give more information. |
| 306 | +The [bromite guide] has been instrumental in handling the authentication. |
| 307 | + |
| 308 | +Another is [goma and buildgrid], which sets up a service account to work with the authentication, |
| 309 | +rather than patching it away. |
| 310 | + |
| 311 | +[depot tools]: https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up |
| 312 | + |
| 313 | +[goma and buildgrid]: https://kubala.github.io/docs/setting-up-goma |
| 314 | +[build chromium]: https://chromium.googlesource.com/infra/goma/client#how-to-use |
| 315 | +[additional building info]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md |
| 316 | + |
| 317 | +## Technical notes for the rbe proxy |
| 318 | + |
| 319 | +### Instance Name |
| 320 | + |
| 321 | +The instance name is handled as a path segment, |
| 322 | +so the empty instance name typically used will be converted to a dot ".". |
| 323 | +So you cannot setup `Buildbarn` to have an empty instance name. |
0 commit comments