object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found

[daviewales]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, object_store automatically detects Azure credentials from environment variables and IMDS. object_store also supports retrieving credentials from Azure CLI, however, it does not automatically try this.
If you want to retrieve credentials from Azure CLI, you need to manually specify use_azure_cli=True.

Describe the solution you'd like

Ideally, object_store would try use_azure_cli=True if no environment or IMDS credentials can be found. This makes it possible to easily test code on your local machine (based on your Azure CLI credentials), then deploy the code unchanged to an Azure VM, and it will automatically use the environment or IMDS credentials, without requiring any code changes.

Describe alternatives you've considered

A workaround is to add AZURE_USE_AZURE_CLI=True to your local environment variables while testing. However, this is an extra step.

Additional context

Most Azure libraries fall back to the Azure CLI credential if environment or IMDS credentials are not found. For example, fsspec/adlfs, azure.identity.DefaultAzureCredential, polars, pandas.

I'm aware of #6470, but I'm wondering if simply adding a fallback to an already supported authentication method is more in scope than completely replicating DefaultAzureCredential.

Related to this downstream obstore issue: developmentseed/obstore#267 (comment)

[kylebarron]

Thanks for creating this issue. I don't know enough about Azure to argue for or against this.

I assume this would be considered a breaking change, and so it would need a resolution within the next few days (unless we want to wait a while).

[tustvold]

My 2 cents is I would not be expecting an application to be shelling out unless I told it to do so, especially given IIRC the azure binary doesn't have a very collision resistant name. I'm not familiar enough with Azure tooling to know if this is an unusual expectation.

That being said, applications wanting this behaviour by default could override the default options

[kylebarron]

I think that's a fair rationale for keeping the default here as false.

While I want the Python bindings to minimize the differences from object_store for best maintainability, I think this is a place where it might be reasonable to override the default on the application side for easy usability for Python end users. developmentseed/obstore#292

[daviewales]

Thanks. I understand the hesitation to shell out to Azure CLI. On the other hand, this is a common behaviour of many Azure tools.

Looking at the code, I think that the only Azure CLI command used is az account get-access-token.
If shelling out is the only concern, it should be possible to inspect the tokens stored in $AZURE_CONFIG_DIR/accessTokens.json, where $AZURE_CONFIG_DIR is ~/.azure by default.

This article notes that this file contains both the latest "accessToken", (60 minute expiry) and also a long lived (90 day) "refreshToken".

If the access token is expired, it should be possible to POST the refresh token to the "_authority" to get a fresh access token. For example, if the "_authority" is https://login.microsoftonline.com/common, then you should be able to get a new 60 minute access token by POSTing the refresh token to https://login.microsoftonline.com/common/oauth2/v2.0/token. Other parameters from the accessTokens.json may also be required. See this SO post for an example.

[kylebarron]

That being said, applications wanting this behaviour by default could override the default options

I implemented this in obstore in developmentseed/obstore#292, however I had initially expected that it would try the Azure CLI, and if it didn't exist or couldn't access credentials, then it would still try instance metadata credentials.

But rereading

arrow-rs/object_store/src/azure/builder.rs

Lines 1015 to 1028 in 95c42a7

	} else if self.use_azure_cli.get()? {
	Arc::new(AzureCliCredential::new()) as _
	} else {
	let msi_credential = ImdsManagedIdentityProvider::new(
	self.client_id,
	self.object_id,
	self.msi_resource_id,
	self.msi_endpoint,
	);
	Arc::new(TokenCredentialProvider::new(
	msi_credential,
	http.connect(&self.client_options.metadata_options())?,
	self.retry_config.clone(),
	)) as _

shows that I was wrong; if use_azure_cli is set to True by default, then instance credentials will never be checked unless the end user explicitly sets use_azure_cli to False