Recommended ways to add resource tags to tests to limit parallelism?

[Ryang20718]

On local execution, we can add resource tags for each test to help reduce likelihood of bazel running out of resources.

Assuming we have 100 tests tagged with the following,

resources:gpu_memory_mb:2000,
cpu:3,
resources:memory:1900

By specifying test --local_resources=gpu_memory_mb=16000 --local_resources=memory=HOST_RAM*0.6 if we have a CI runner with 48 vcpus, 1 16 GB gpu, 96 GB of ram executing tests locally, we can limit test parallelism scheduling so that we can fully utilize the runner whilst still not running out of resources.

Is there a similar way to do this via buildbarn? I know the Initial Size Class Cache exists so that users don't need to maintain this test tag heuristic. However, for users that have this data, is there any way to leverage this data in buildbarn to limit concurrency to avoid Out of memory errors?

😅 I guess this is mainly a cost optimization question to maximize machine utilization whilst minimizing instabilities.