az storage azcopy blob upload fails with Azurite emulator

[flcdrg]

Describe the bug

Tried to use az storage azcopy blob upload to upload a file to Azurite

Related command

az storage azcopy blob upload --container "licensemanagement" --connection-string "AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;DefaultEndpointsProtocol=http;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;" --source "C:\MyFile.pdf" --destination "MyFile.pdf"

Errors

Azcopy command: ['C:\Users\user\.Azure\cliextensions\storage-preview\azext_storage_preview\azcopy\azcopy_windows_amd64_10.5.0\azcopy.exe', 'copy', 'C:\Users\user\Downloads\User Guide - Microsoft Authenticator.pdf', 'https://devstoreaccount1.blob.core.windows.net/licensemanagement/auth.pdf?se=2024-02-02T00%3A13%3A14Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=E2ES3N2469Zdf9cBKuSH9aa%2BLAkr8PGoGZk/dK78r4k%3D']
INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 770ee4b7-0305-cc4a-68ec-5c0053799eec has started
Log file is located at: C:\Users\user.azcopy\770ee4b7-0305-cc4a-68ec-5c0053799eec.log

INFO: azcopy.exe: A newer version 10.23.0 is available to download

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:4c668ee1-a01e-006a-18a3-546a7d000000
Time:2024-02-01T00:13:17.1966259Z, Details:
AuthenticationErrorDetail: Signature did not match. String to sign used was devstoreaccount1
rwdlacup
b
sco

2024-02-02T00:13:14Z

2018-03-28

Code: AuthenticationFailed
PUT https://devstoreaccount1.blob.core.windows.net/licensemanagement/auth.pdf?se=2024-02-02t00%3A13%3A14z&sig=-REDACTED-&sp=rwdlacup&srt=sco&ss=b&sv=2018-03-28&timeout=901
Content-Length: [577644]
User-Agent: [AzCopy/10.5.0 Azure-Storage/0.10 (go1.13; Windows_NT)]
X-Ms-Blob-Cache-Control: []
X-Ms-Blob-Content-Disposition: []
X-Ms-Blob-Content-Encoding: []
X-Ms-Blob-Content-Language: []
X-Ms-Blob-Content-Md5: []
X-Ms-Blob-Content-Type: [application/pdf]
X-Ms-Blob-Type: [BlockBlob]
X-Ms-Client-Request-Id: [bb5090a1-fc6c-48b9-7522-292781581b0c]
X-Ms-Version: [2019-02-02]

RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Content-Length: [491]
Content-Type: [application/xml]
Date: [Thu, 01 Feb 2024 00:13:16 GMT]
Server: [Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [AuthenticationFailed]
X-Ms-Request-Id: [4c668ee1-a01e-006a-18a3-546a7d000000]

100.0 %, 0 Done, 1 Failed, 0 Pending, 0 Skipped, 1 Total,

Job 770ee4b7-0305-cc4a-68ec-5c0053799eec summary
Elapsed Time (Minutes): 0.1
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Total Number of Transfers: 1
Number of Transfers Completed: 0
Number of Transfers Failed: 1
Number of Transfers Skipped: 0
TotalBytesTransferred: 0
Final Job Status: Cancelled

Issue script & Debug output

DEBUG: cli.knack.cli: Command arguments: ['storage', 'azcopy', 'blob', 'upload', '--container', 'licensemanagement', '--connection-string', 'AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;DefaultEndpointsProtocol=http;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;', '--source', 'C:\Users\user\Downloads\User Guide - Microsoft Authenticator.pdf', '--destination', 'auth.pdf', '--debug']
DEBUG: cli.knack.cli: init debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x0000016D5D7DF560>, <function OutputProducer.on_global_arguments at 0x0000016D5D969C60>, <function CLIQuery.on_global_arguments at 0x0000016D5D9937E0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'storage': ['azext_storage_preview', 'azure.cli.command_modules.storage']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: storage 0.215 59 273
DEBUG: cli.azure.cli.core: Total (1) 0.215 59 273
DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory
DEBUG: cli.knack.cli: Event: CommandLoader.OnLoadCommandTable []
DEBUG: cli.azure.cli.core: storage-preview 0.022 16 52 C:\Users\user.azure\cliextensions\storage-preview
DEBUG: cli.azure.cli.core: Total (1) 0.022 16 52
DEBUG: cli.azure.cli.core: Loaded 65 groups, 295 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : storage azcopy blob upload
DEBUG: cli.azure.cli.core: Command table: storage azcopy blob upload
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x0000016D5F01A160>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\user.azure\commands\2024-02-01.10-47-16.storage_azcopy_blob_upload.7176.log'.
INFO: az_command_data_logger: command args: storage azcopy blob upload --container {} --connection-string {} --source {} --destination {} --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x0000016D5F052480>]
DEBUG: cli.knack.cli: Event: CommandLoader.OnLoadArguments []
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 659, in _get_attr
AttributeError: module 'azext_storage_preview.vendored_sdks.azure_mgmt_storage.v2022_09_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'

DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x0000016D5F080540>, <function register_cache_arguments..add_cache_arguments at 0x0000016D5F080680>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0000016D5D969D00>, <function CLIQuery.handle_query_parameter at 0x0000016D5D993880>, <function register_ids_argument..parse_ids_arguments at 0x0000016D5F0805E0>]
INFO: az_command_data_logger: extension name: storage-preview
INFO: az_command_data_logger: extension version: 1.0.0b1
WARNING: Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
DEBUG: cli.azext_storage_preview._client_factory: Getting data service client service_type=BlockBlobService
DEBUG: cli.azext_storage_preview._client_factory: Getting data service client service_type=BlockBlobService
WARNING: cli.azext_storage_preview.azcopy.util: Azcopy command: ['C:\Users\user\.Azure\cliextensions\storage-preview\azext_storage_preview\azcopy\azcopy_windows_amd64_10.5.0\azcopy.exe', 'copy', 'C:\Users\user\Downloads\User Guide - Microsoft Authenticator.pdf', 'https://devstoreaccount1.blob.core.windows.net/licensemanagement/auth.pdf?se=2024-02-02T00%3A17%3A16Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=QcZYEOGaTiQwKqnnn4FJzatT524kjGk443ARpTJ07KQ%3D']
DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x0000016D5F0516C0>, <function x509_from_base64_to_hex_transform at 0x0000016D5F051760>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x0000016D5F01A3E0>]
INFO: az_command_data_logger: exit code: 0
INFO: cli.main: Command ran in 7.095 seconds (init: 0.445, invoke: 6.650)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3466 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry_init.pyc C:\Users\user.azure"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process.

Expected behavior

Should be able to upload to Azurite

Environment Summary

azure-cli 2.56.0

core 2.56.0
telemetry 1.1.0

Extensions:
application-insights 1.0.0
azure-devops 0.26.0
managementpartner 0.1.3
storage-preview 1.0.0b1

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\user.azure\cliextensions'

Python (Windows) 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @flcdrg, wondering how the connection string was generated, seems like authentication problem, maybe the signature is malformed? Can you try the command az storage blob upload with the same connection string as well? Thanks.

[flcdrg]

The connection string was copied from the "Primary Connection String" property in Microsoft Azure Storage Explorer (pointing at the Emulator).

Yes, az storage blob upload works correctly with the same connection string

[calvinhzy]

Hi @flcdrg The az storage azcopy is a very old preview command group which has been replaced with either az storage blob which uses the Python SDK and az storage copy which uses azcopy as the underlying application. Please give either one a try. az storage blob upload should cover most use cases for az storage azcopy blob upload. Thanks