Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codebuild fails due to lack of resources #9088

Closed
xmfcx opened this issue Oct 15, 2024 · 4 comments
Closed

Codebuild fails due to lack of resources #9088

xmfcx opened this issue Oct 15, 2024 · 4 comments
Assignees
Labels
status:stale Inactive or outdated issues. (auto-assigned) type:ci Continuous Integration (CI) processes and testing. (auto-assigned)

Comments

@xmfcx
Copy link
Contributor

xmfcx commented Oct 15, 2024

Also:
image

https://github.com/autowarefoundation/autoware.universe/actions/runs/11341739442/job/31550579344

@mitsudome-r these are still managing to fail. Let's try to find a solution in the Software WG meeting.

Probably failing because of lack of memory.
These are CodeBuild machines come pre-set for GitHub actions.
I couldn't find a way to assign swap to these machines.

Current options:

  • Limit the number of parallel jobs without using taskset?
  • Reduce the cpus assigned to taskset?
  • Current codebuild cost is about $500 per month, maybe just find another place to simply host about 10 dedicated machines and set up github actions runners on them?

Originally posted by @xmfcx in #8946 (comment)

@xmfcx xmfcx self-assigned this Oct 15, 2024
@xmfcx xmfcx added the type:ci Continuous Integration (CI) processes and testing. (auto-assigned) label Oct 15, 2024
@xmfcx xmfcx moved this to Todo in Software Working Group Oct 15, 2024
@xmfcx
Copy link
Contributor Author

xmfcx commented Oct 15, 2024

Related branch by @mitsudome-r

@xmfcx xmfcx changed the title Codebuild failing due to lack of resources Codebuild fails due to lack of resources Oct 15, 2024
@mitsudome-r
Copy link
Member

The only solution that worked so far was reducing the number of cpu cores used for build:
https://github.com/autowarefoundation/autoware.universe/actions/runs/11345123914

Some other options other that what Fatih mentioned could be:

  • find out a root cause why Autoware build requires so much memory
  • build the packages that require a lot of memory separately (if we can find out a way to manipulate build order)

Copy link

stale bot commented Dec 19, 2024

This pull request has been automatically marked as stale because it has not had recent activity.

@stale stale bot added the status:stale Inactive or outdated issues. (auto-assigned) label Dec 19, 2024
@xmfcx
Copy link
Contributor Author

xmfcx commented Dec 23, 2024

This was fixed by using less cores while building on codebuild.

We'll go back to public runners since they provide about 10GB more space on 24.04 runners.

@xmfcx xmfcx closed this as completed Dec 23, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Software Working Group Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status:stale Inactive or outdated issues. (auto-assigned) type:ci Continuous Integration (CI) processes and testing. (auto-assigned)
Projects
Archived in project
Development

No branches or pull requests

2 participants