Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use production propolis-server flags in PHD CI builds #878

Merged
merged 7 commits into from
Mar 8, 2025
Merged

Conversation

iximeow
Copy link
Member

@iximeow iximeow commented Mar 4, 2025

We use the dev (e.g. non-release) build of propolis-server in PHD to get debug assertions, which we want, but we should use the omicron-build feature to be in line with production binaries.

omicron-build had also acquired code related to failure injection, which we also want in CI so we can test migration failure codepaths. That is moved to a new failure-injection feature flag with a bit of ceremony to stop us early if we're ever inadvertently packaging a propolis-server with omicron-build and failure-injection.

Fixes #412. I assume the failure-injection feature will be useful for future testing enhancements like #698

cargo xtask phd run still passes with an Alpine guest, waiting to see the same from phd-run.

We use the dev (e.g. non-release) build of propolis-server in PHD to get
debug assertions, which we want, but we should use the `omicron-build`
feature to be in line with production binaries.

`omicron-build` had also acquired code related to failure injection,
which we *also* want in CI so we can test migration failure codepaths.
That is moved to a new `failure-injection` feature flag with a bit of
ceremony to stop us early if we're ever inadvertently packaging a
propolis-server with `omicron-build` and `failure-injection`.

Fixes #412.
@iximeow iximeow marked this pull request as ready for review March 4, 2025 21:41
@iximeow
Copy link
Member Author

iximeow commented Mar 4, 2025

all the failures include propolis-server stderr like "error_message_internal":"VM initialization failed: failed to join VM object creation task: failed to add low memory region"

there are info logs for VMM allocations and from a passing build we have been saying reservoir too small (0MiB) to use for guest memory. so, now with omicron-build it's unconditionally trying and failing to use the zero-size reservoir, and every test fails. onward to wonder what's up with buskin.

@gjcolombo gjcolombo self-requested a review March 6, 2025 19:08
Copy link
Contributor

@gjcolombo gjcolombo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple little nits, but overall this looks good to me. Thanks for working on this!

// in the system has encountered an error, so enable
// failure-injection. Do not enable `omicron-build` like we
// do in Buildomat because someone running `cargo xtask phd`
// is not building a propolis-server destined for Omicron.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, unless they are :) I think this was my use case earlier this week, and it may also come in handy if someone wants to try to reproduce a CI failure on a local machine.

Perhaps add an omicron_build switch to PropolisArgs to allow the user to opt into adding that feature?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

..... i suppose cargo xtask phd does build a replacement propolis-server doesn't it.

but yeah, a build to get omicron-build enabled makes a lot of sense. lemme do that.

Copy link
Member Author

@iximeow iximeow Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in writing 8affc65 i discovered that cargo xtask phd run and cargo xtask phd run --omicron-build ends up with artifacts in different directories. so if you run one, then the other, then the first one again expecting another propolis-server rebuild, it'll actually just run immediately. surprise!

but the test suite passes on my workstation with or without the flag (wherein i have a reservoir configured), everything fails (as expected!) when i set the reservoir too small.

@iximeow iximeow merged commit 9ee2e1f into master Mar 8, 2025
11 checks passed
@iximeow iximeow deleted the ixi/412 branch March 8, 2025 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Set up PHD runs using binaries with production feature flags
3 participants