Skip to content

Commit 4d4ec28

Browse files
stagnationmoroten
stagnation
authored and
moroten
committed
Document how to use goma with Buildbarn
A summary of the steps taken in this feature request: buildbarn#96
1 parent 65ec42d commit 4d4ec28

File tree

2 files changed

+328
-0
lines changed

2 files changed

+328
-0
lines changed

‎README.md

+5
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,11 @@ remote_execution_extra_platform_properties = [
118118
]
119119
```
120120

121+
### Goma
122+
123+
It is possible to use `goma`, the buildsystem in the `chromium` project with `Buildbarn`.
124+
The instructions are available here: [docs/goma.md].
125+
121126
### Bazel without a remote toolchain
122127

123128
You do not need to define a toolchain for remote execution, like this repository does.

‎docs/goma.md

+323
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,323 @@
1+
# Goma and Buildbarn
2+
3+
This is an incomplete overview of the steps needed to run `goma` with `Buidlbarn`.
4+
It is based on the [discussion in the issue] as well as private chats.
5+
Big thanks to [Jidong Qin](https://github.com/qinjidong) for figuring it all out! 🎉
6+
7+
[discussion in the issue]: https://github.com/buildbarn/bb-deployments/issues/96
8+
9+
## Setup
10+
11+
1) Install google's [depot tools] to manage dependencies
12+
and working with `goma`.
13+
These are needed on PATH, but contains a lot of scripts.
14+
So it is best to keep them isolated and only add them to PATH
15+
in the terminal where you work with `goma`.
16+
17+
2) Install [goma client].
18+
We use `cipd` which comes from `depot tools`:
19+
20+
$ cipd install infra/goma/client/linux-amd64 -root ~/goma
21+
22+
3) Checkout the [goma server].
23+
24+
3) We need something to build, them most common is the `chromium` project.
25+
With documentation available here: [build chromium] and [additional building info].
26+
27+
[goma client]: https://chromium.googlesource.com/infra/goma/client/
28+
[goma server]: https://chromium.googlesource.com/infra/goma/server/
29+
30+
## Overview
31+
32+
There are a few parts in this,
33+
`ninja` is the main buildsystem, which calls `goma`,
34+
and a compiler wrapper, `gomacc`.
35+
36+
The `goma` client will itself spin up two background tasks:
37+
a `http_proxy` that connects the client to the server,
38+
you'll notice that the ports used for the client point to this proxy,
39+
which in-turn talks to the `goma` server, `rbe proxy`.
40+
41+
The `compiler proxy` too is central in this,
42+
it includes a detailed web-page for all compile actions,
43+
and their errors as well as server logs (info, warn, error).
44+
45+
ninja
46+
47+
-> goma client
48+
-> http_proxy
49+
-> compiler_proxy
50+
51+
-> goma server (rbe proxy)
52+
-> Buildbarn
53+
54+
## Patches
55+
56+
`goma` is designed to work with Google infrastructure
57+
and authentication.
58+
We do not use either, but there are no feature flags for this behavior,
59+
so we must patch three components.
60+
61+
### Patch the client
62+
63+
We cut off the authentication code in `goma_auth.py`
64+
I did it in the client repository, but it is easier to just install with `cipd`.
65+
66+
67+
goma $ git diff
68+
diff --git a/client/goma_auth.py b/client/goma_auth.py
69+
index 5cc674d..e5425e9 100755
70+
--- a/client/goma_auth.py
71+
+++ b/client/goma_auth.py
72+
@@ -1,574 +1,5 @@
73+
#!/usr/bin/env python3
74+
75+
-# Copyright 2015 The Goma Authors. All rights reserved.
76+
-if __name__ == '__main__':
77+
...
78+
- sys.exit(main())
79+
+ print("Bypassing authentication 'goma_auth.py'.")
80+
+ return 0
81+
82+
83+
84+
$ cat ~/goma/bin/goma_auth.py
85+
#!/usr/bin/env python3
86+
87+
def main():
88+
print("Bypassing authentication 'goma_auth.py'.")
89+
return 0
90+
91+
### Patch the remote execution proxy
92+
93+
First, to work with `Buildbarn` we need a simple patch,
94+
patch the `OSFamily` platform property to lowercase.
95+
In the `goma` server repository:
96+
97+
commit 8d1ba1eb6aed0b504448f464ae365e9af705788c (HEAD)
98+
Author: Nils Wireklint <nils@meroton.com>
99+
Date: Tue Aug 15 11:50:19 2023 +0200
100+
101+
Fix OSFamily value capitalization
102+
103+
In accordance with the REv2 API the standard value of the OSFamily
104+
platform property should be lowercase.
105+
106+
See
107+
https://github.com/bazelbuild/remote-apis/blob/068363a3625e166056c155f6441cfb35ca8dfbf2/build/bazel/remote/execution/v2/platform.md
108+
109+
diff --git a/cmd/remoteexec_proxy/main.go b/cmd/remoteexec_proxy/main.go
110+
index 4ab92a2..d321344 100644
111+
--- a/cmd/remoteexec_proxy/main.go
112+
+++ b/cmd/remoteexec_proxy/main.go
113+
@@ -412,7 +412,7 @@ func main() {
114+
Value: *platformContainerImage,
115+
}, {
116+
Name: "OSFamily",
117+
- Value: "Linux",
118+
+ Value: "linux",
119+
},
120+
},
121+
},
122+
{
123+
124+
Run the `goma` server (to proxy to RBE):
125+
126+
goma/server $ go run \
127+
cmd/remoteexec_proxy/main.go \
128+
-port 5050 \
129+
-remoteexec-addr localhost:8980 \
130+
-remote-instance-name "hardlinking" \
131+
-platform-container-image 'docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448' \
132+
-insecure-remoteexec
133+
134+
The container image platform property is set on the command line.
135+
This should say that it is running, accepts you and can talk RBE:
136+
137+
2023-08-17T11:33:40.842+0200 INFO exec/inventory.go:190 configure platform config: target:{addr:"grpc://127.0.0.1:8980"} build_info:{} remoteexec_platform:{properties:{name:"container-image" value:"docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448"} properties:{name:"OSFamily" value:"linux"} rbe_instance_basename:"hardlinking"} dimensions:"os:linux"
138+
139+
### Patch the goma server
140+
141+
The majority of authentication and access token handling is done in the server,
142+
so we can patch away all of that.
143+
Following the instructions from the [bromite guide]
144+
145+
First create a dummy token file:
146+
147+
echo "nomatter" > ~/.debug_goma_auth_file
148+
149+
Then patch the `goma server`:
150+
151+
commit f9365d8432ad8e4cb8832b19dcec1a0301f634f8 (HEAD)
152+
Author: Nils Wireklint <nils@meroton.com>
153+
Date: Fri Aug 18 12:25:50 2023 +0200
154+
155+
apply patch from bromite/discussions/1032
156+
157+
diff --git a/auth/acl/checker.go b/auth/acl/checker.go
158+
index f225309..f70edd3 100644
159+
--- a/auth/acl/checker.go
160+
+++ b/auth/acl/checker.go
161+
@@ -114,6 +114,8 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo
162+
163+
logger := log.FromContext(ctx)
164+
165+
+ return "id1", nil, nil
166+
+
167+
g, err := c.FindGroup(ctx, tokenInfo)
168+
if err != nil {
169+
if ctx.Err() != nil {
170+
@@ -163,6 +165,7 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo
171+
func checkGroup(ctx context.Context, tokenInfo *auth.TokenInfo, g *pb.Group, authDB AuthDB) (bool, error) {
172+
logger := log.FromContext(ctx)
173+
logger.Debugf("checking group:%s", g.Id)
174+
+ return true, nil
175+
if g.Audience != "" {
176+
if tokenInfo.Audience != g.Audience {
177+
logger.Debugf("audience mismatch: %s != %s", tokenInfo.Audience, g.Audience)
178+
diff --git a/auth/client.go b/auth/client.go
179+
index 1e49076..8e5ca09 100644
180+
--- a/auth/client.go
181+
+++ b/auth/client.go
182+
@@ -153,6 +153,12 @@ func (a *Auth) Check(ctx context.Context, req *http.Request) (*enduser.EndUser,
183+
defer span.End()
184+
logger := log.FromContext(ctx)
185+
186+
+ fake_token := &oauth2.Token{
187+
+ AccessToken: "",
188+
+ TokenType: "",
189+
+ }
190+
+ return enduser.New("fake_email", "fake_groupid", fake_token), nil
191+
+
192+
authorization := req.Header.Get("Authorization")
193+
if authorization == "" {
194+
logger.Warnf("no authorization header")
195+
diff --git a/remoteexec/exec.go b/remoteexec/exec.go
196+
index 251b5e9..a5515d8 100644
197+
--- a/remoteexec/exec.go
198+
+++ b/remoteexec/exec.go
199+
@@ -604,6 +604,7 @@ func (r *request) newInputTree(ctx context.Context) *gomapb.ExecResp {
200+
}
201+
202+
symAbsOk := r.f.capabilities.GetCacheCapabilities().GetSymlinkAbsolutePathStrategy() == rpb.SymlinkAbsolutePathStrategy_ALLOWED
203+
+ symAbsOk = true
204+
205+
cmdCleanCWD := cleanCWD
206+
cmdCleanRootDir := cleanRootDir
207+
:
208+
commit f9365d8432ad8e4cb8832b19dcec1a0301f634f8 (HEAD)
209+
Author: Nils Wireklint <nils@meroton.com>
210+
Date: Fri Aug 18 12:25:50 2023 +0200
211+
212+
apply patch from bromite/discussions/1032
213+
214+
diff --git a/auth/acl/checker.go b/auth/acl/checker.go
215+
index f225309..f70edd3 100644
216+
--- a/auth/acl/checker.go
217+
+++ b/auth/acl/checker.go
218+
@@ -114,6 +114,8 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo
219+
220+
logger := log.FromContext(ctx)
221+
222+
+ return "id1", nil, nil
223+
+
224+
g, err := c.FindGroup(ctx, tokenInfo)
225+
if err != nil {
226+
if ctx.Err() != nil {
227+
@@ -163,6 +165,7 @@ func (c *Checker) CheckToken(ctx context.Context, token *oauth2.Token, tokenInfo
228+
func checkGroup(ctx context.Context, tokenInfo *auth.TokenInfo, g *pb.Group, authDB AuthDB) (bool, error) {
229+
logger := log.FromContext(ctx)
230+
logger.Debugf("checking group:%s", g.Id)
231+
+ return true, nil
232+
if g.Audience != "" {
233+
if tokenInfo.Audience != g.Audience {
234+
logger.Debugf("audience mismatch: %s != %s", tokenInfo.Audience, g.Audience)
235+
diff --git a/auth/client.go b/auth/client.go
236+
index 1e49076..8e5ca09 100644
237+
--- a/auth/client.go
238+
+++ b/auth/client.go
239+
@@ -153,6 +153,12 @@ func (a *Auth) Check(ctx context.Context, req *http.Request) (*enduser.EndUser,
240+
defer span.End()
241+
logger := log.FromContext(ctx)
242+
243+
+ fake_token := &oauth2.Token{
244+
+ AccessToken: "",
245+
+ TokenType: "",
246+
+ }
247+
+ return enduser.New("fake_email", "fake_groupid", fake_token), nil
248+
+
249+
authorization := req.Header.Get("Authorization")
250+
if authorization == "" {
251+
logger.Warnf("no authorization header")
252+
diff --git a/remoteexec/exec.go b/remoteexec/exec.go
253+
index 251b5e9..a5515d8 100644
254+
--- a/remoteexec/exec.go
255+
+++ b/remoteexec/exec.go
256+
@@ -604,6 +604,7 @@ func (r *request) newInputTree(ctx context.Context) *gomapb.ExecResp {
257+
}
258+
259+
symAbsOk := r.f.capabilities.GetCacheCapabilities().GetSymlinkAbsolutePathStrategy() == rpb.SymlinkAbsolutePathStrategy_ALLOWED
260+
+ symAbsOk = true
261+
262+
cmdCleanCWD := cleanCWD
263+
264+
265+
And start the `goma server`:
266+
267+
~/goma/goma_ctl.py start
268+
269+
[bromite guide]: https://github.com/bromite/bromite/discussions/1032
270+
271+
## Setup Chromium
272+
273+
Download and skip the history:
274+
275+
$ fetch --nohooks --no-history chromium
276+
$ cd src
277+
$ ./build/install-build-deps.sh
278+
$ gclient sync
279+
280+
$ gn args out/Default
281+
282+
## Building Chromium
283+
284+
chromium/src $ export GOMA_SERVER_HOST=localhost
285+
export GOMA_SERVER_PORT=5050
286+
export GOMA_USE_SSL=false
287+
export GOMA_HERMETIC=error
288+
export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true
289+
export GOMA_HTTP_AUTHORIZATION_FILE=~/.debug_goma_auth_file
290+
export GOMA_USE_LOCAL=false
291+
export GOMA_FALLBACK=true
292+
293+
# generate ninja files to use goma
294+
$ gn gen --args="use_goma=true goma_dir=\"~/goma\" " out/Default
295+
296+
$ ~/goma/goma_ctl.py start
297+
298+
# build something
299+
$ ninja -j16 -C out/Default obj/base/base/base64.o
300+
301+
# Appendix
302+
303+
## Other guides
304+
305+
There are a few other guides that can give more information.
306+
The [bromite guide] has been instrumental in handling the authentication.
307+
308+
Another is [goma and buildgrid], which sets up a service account to work with the authentication,
309+
rather than patching it away.
310+
311+
[depot tools]: https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up
312+
313+
[goma and buildgrid]: https://kubala.github.io/docs/setting-up-goma
314+
[build chromium]: https://chromium.googlesource.com/infra/goma/client#how-to-use
315+
[additional building info]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md
316+
317+
## Technical notes for the rbe proxy
318+
319+
### Instance Name
320+
321+
The instance name is handled as a path segment,
322+
so the empty instance name typically used will be converted to a dot ".".
323+
So you cannot setup `Buildbarn` to have an empty instance name.

0 commit comments

Comments
 (0)