Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: mirror filecoin built-in actors #344

Merged
merged 63 commits into from
Jan 24, 2024

Conversation

samuelarogbonlo
Copy link
Contributor

@samuelarogbonlo samuelarogbonlo commented Nov 16, 2023

Summary of changes
Changes introduced in this pull request:

  • added a workflow running every hour.
  • Downloads filecoin built-in actors released in the last two years
  • Checks if we need to do any mirroring by comparing downloaded actors byte for byte with the ones in the object storage(R2 and DO)
  • If yes, upload the missing assets.

Reference issue to close (if applicable)

Closes #323

Other information and links

samuelarogbonlo and others added 15 commits November 16, 2023 13:45
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@samuelarogbonlo samuelarogbonlo marked this pull request as ready for review November 16, 2023 15:25
@samuelarogbonlo samuelarogbonlo requested a review from a team as a code owner November 16, 2023 15:25
@samuelarogbonlo samuelarogbonlo requested review from ruseinov, elmattic and LesnyRumcajs and removed request for a team November 16, 2023 15:25
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Copy link
Member

@LesnyRumcajs LesnyRumcajs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mirroring only the latest release is not useful. The goal is to mirror all releases, properly versioned. The end structure of the mirror should be like (though more complete):

├── v10.0.0
│   ├── builtin-actors-devnet.car
│   └── builtin-actors-mainnet.car
├── v10.0.0-rc.1
│   └── builtin-actors-calibrationnet.car
├── v11.0.0
│   ├── builtin-actors-devnet.car
│   └── builtin-actors-mainnet.car

@samuelarogbonlo samuelarogbonlo marked this pull request as draft November 24, 2023 10:45
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@samuelarogbonlo samuelarogbonlo marked this pull request as ready for review December 1, 2023 13:41
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@ruseinov
Copy link

Looks good, however since we're running this via cron - why don't we also make sure that dependencies are cahched? Here's a guide.

@ruseinov done

How about caching the installation itself: https://github.com/snok/install-poetry?tab=readme-ov-file#caching-the-poetry-installation

There are also useful guides on how to cache deps in the above readme too.

Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@samuelarogbonlo
Copy link
Contributor Author

Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@LesnyRumcajs
Copy link
Member

@samuelarogbonlo I appreciate the effort of making it enterprise-grade, but a simple lambda would do enough for having a configurable sink. We're getting into non-trivial Python code where we shouldn't.

Pseudocode of what I mean:

sink = if use_local
  || save_to_file(file)
else
  || save_to_s3(file)
end

You also added quite a bit of unused methods. Let's keep things as simple as possible.

@ruseinov
Copy link

@samuelarogbonlo I appreciate the effort of making it enterprise-grade, but a simple lambda would do enough for having a configurable sink. We're getting into non-trivial Python code where we shouldn't.

Pseudocode of what I mean:

sink = if use_local
  || save_to_file(file)
else
  || save_to_s3(file)
end

You also added quite a bit of unused methods. Let's keep things as simple as possible.

I think there is exactly one unused method: exists. What you suggest would work, if we didn't need list_files that has quite a bit of logic. And just like that one lambda is not sufficient anymore, unless we want to duplicate the logic the two sinks otherwise share.

@LesnyRumcajs
Copy link
Member

If you guys are okay with that, then I won't block it.

@LesnyRumcajs LesnyRumcajs dismissed their stale review January 23, 2024 17:16

reviewed by others

samuelarogbonlo and others added 6 commits January 23, 2024 18:30
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@samuelarogbonlo
Copy link
Contributor Author

If you guys are okay with that, then I won't block it.

@LesnyRumcajs. Good?

samuelarogbonlo and others added 3 commits January 24, 2024 16:06
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
Signed-off-by: samuelarogbonlo <sbayo971@gmail.com>
@samuelarogbonlo samuelarogbonlo enabled auto-merge (squash) January 24, 2024 15:13
@samuelarogbonlo samuelarogbonlo merged commit 690e99e into main Jan 24, 2024
10 checks passed
@samuelarogbonlo samuelarogbonlo deleted the samuel/mirror-builtin-actors-releases branch January 24, 2024 15:14
@ruseinov
Copy link

ruseinov commented Jan 24, 2024

If you guys are okay with that, then I won't block it.

I kinda get where you are coming from, but I don't see a better way here. This is not too advanced, just using some OOP that happens to be quite ugly in Python :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mirror builtin-actors releases
4 participants