Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PBM-1484 GCP SDK #1096

Merged
merged 66 commits into from
Feb 24, 2025
Merged

PBM-1484 GCP SDK #1096

merged 66 commits into from
Feb 24, 2025
+250,883 −0

Conversation

veceraj
Copy link
Collaborator

@veceraj veceraj commented Feb 19, 2025 •
edited
Loading

PR for https://perconadev.atlassian.net/browse/PBM-1484

Migration from AWS SDK v1 to v2 introduced unexpected issues with Google Cloud Storage.
This feature implements the Google Cloud Storage package.

Credentials

Example storage config:

storage:
  type: gcs
  gcs:
    bucket: pbm-bcp
    prefix: pbm/test
    credentials:
      projectId: psychic-mason-44...
      privateKey: -----BEGIN PRIVATE KEY-----\n ...

ChunkSize

storage.gcs.chunkSize (optional) specified in bytes (https://pkg.go.dev/cloud.google.com/go/storage#Writer)

  • defaults to nil - the GCS default is 16MB
  • it is possible to specify 0
    • From GCS docs: "If ChunkSize is set to zero, chunking will be disabled and the object will be uploaded in a single request without the use of a buffer."
  • (Note: s3 has uploadPartSize, there is no equivalent to maxUploadParts)

veceraj and others added 30 commits January 22, 2025 23:08
@veceraj
use v2 sdk - wip
ee25aa4
@veceraj
migrate download
e65ffa5
@veceraj
update
e85cd46
@veceraj
change mod
d4ccc3d
@veceraj
update version in compress
67d5c3b
@veceraj
remove v1
98b1949
@veceraj
update vendor
50446f6
@veceraj
use ForcePathStyle
2853a2a
@veceraj
cleanup
4ce09bf
@veceraj
check not found
b28103c
@veceraj
update doc for delete
66f91f5
@veceraj
implement logger
e064043
@veceraj
update go mod
c1cf838
@veceraj
update status
bd77e52
@veceraj
Merge branch 'dev' of https://github.com/percona/percona-backup-mongodb…
0d31425 
… into PBM-1474-aws-s3-migration
@veceraj
rename imports
833db57
@veceraj @github-actions
resolve gofmt
792a1d8 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@veceraj
resolve deprecated
e005f3c
@veceraj
Merge branch 'PBM-1474-aws-s3-migration' of https://github.com/percon…
fc18f23 
…a/percona-backup-mongodb into PBM-1474-aws-s3-migration
@veceraj
cleanup
f6cad0c
@veceraj
resolve gcs bug
83dbbac
@veceraj
wip gcs changes
fc3636f
@veceraj
fix reviewdog
66728a5
@veceraj
fix special chars
3745dc0
@veceraj
fix sse-c
5d4ec2a
@veceraj
add check for AES to copy
3f02f4f
@veceraj
apply fix to download
2ba754a
@veceraj
resolve comments
fa4a615
@veceraj
remove gcs tweaks
dfa699f
@veceraj
revert
4a6c769
@veceraj veceraj marked this pull request as ready for review February 19, 2025 14:43
boris-ilijic
boris-ilijic reviewed Feb 20, 2025
View reviewed changes
Copy link
Member

@boris-ilijic boris-ilijic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with this PR, it's simple and clean. Still we need to do some more improvements, please see the comments.

@boris-ilijic boris-ilijic self-requested a review February 24, 2025 14:48
@veceraj veceraj merged commit 116d50e into dev Feb 24, 2025
20 of 21 checks passed
@veceraj veceraj deleted the PBM-1484-GCP-SDK branch February 24, 2025 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boris-ilijic boris-ilijic boris-ilijic approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@veceraj @boris-ilijic