Feature request: allow user to use a boto3-native credential provider/session for s3 etc.

[jwhitaker-gridcog]

Description

It's nice that Polars has native support for s3, however we have issues with using it with less common credential setups.

In short, I would really like it it I could just rely on boto3 to manage credentials for me - there's a long tail of things that the rust-side object_store seems to struggle with. (for context, when I talk about polars i mean py-polars).

Examples I have hit personally:

• passing profile_name in storage_options (workaround: open a boto client and export with get_frozen_credentials)
• handling assumed roles (workaround: open a boto client, assume the role, export the sts response; or, use an approach like https://github.com/boto/boto3/pull/3253/files to make the boto client assume to role for you + handle refreshing etc).
• you have probably hit more - I guess "making aws creds work like users expect from boto" is a recurring thing that pops up.

In particular, the workarounds are a problem when credentials expire. A boto session/credential handles refreshing for you. However if I'm passing a session token to polars myself via the above hacks, then I have to handle credential expiration myself, which I can't really do in a foolproof way - what happens if I have a long-lived lazyframe loaded with a 15-minute aws session, for example?

If I could just set up a boto3 session myself, and then pass it to polars and say "hey use this", this would be great from a UX perspective.

This could be general enough to go into a how to read from s3 101 doc, which could avoid the traps new users hit ("scan_parquet('s3://...') doesn't work, oh what's this, i have to go setting session tokens myself? ugh").

From your perspective, I wonder if it could simplify things a lot as well - instead of needing to provide an endless set of hooks to configure the rust side "kind of like a boto session", you could literally rely on boto itself.

On the engineering side though, this would require some glue from object_store back up to the python layer when it needs to fetch/refresh credentials - I don't know if this could be done cleanly/acceptably.

[brianrienecker]

+1

[cbartram]

+1

[moorthy156]

+1

[jacoblgoodman]

This would be a huge help +1!

[fdosani]

+1 too

[jkc1]

+1 from me as well

[tustvold]

I've filed #18979 which should allow for this, by exposing the API object_store already has for this

[nameexhaustion]

Closing as the requested functionality is now possible using the credential_provider parameter to the I/O functions. This can accept arbitrary Python functions for retrieving / refreshing credentials.

Available in the current release (1.21.0):

• For assuming a simple role, you can use the built-in pl.CredentialProviderAWS
  • See feat(python): AssumeRole support for AWS Credential Provider #19346 for an example
• For more advanced authentication flows, an arbitrary Python function can be passed as a credential_provider
  • See feat: Experimental credential_provider argument for scan_parquet #19271 for an example

Available in the next release:

• Passing aws_profile in storage_options - feat(python): Support passing aws_profile in storage_options #20965
  • Before then, you can set the AWS_PROFILE environment variable and ensure boto3 is installed. To ensure this is working, set POLARS_VERBOSE=1 in the environment and ensure you observe that CredentialProviderAWS gets initialized in the logs.

[jwhitaker-gridcog]

You guys rock