Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ML Extension causing Azure CLI to be very slow #30794

Closed
cecheta opened this issue Feb 9, 2025 · 11 comments
Closed

ML Extension causing Azure CLI to be very slow #30794

cecheta opened this issue Feb 9, 2025 · 11 comments
Assignees
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. Machine Learning az ml Possible-Solution Service Attention This issue is responsible by Azure service team.

Comments

@cecheta
Copy link
Member

cecheta commented Feb 9, 2025

Describe the bug

Installing the Azure CLI Machine Learning extension makes the CLI extremely slow. The installation of the extension itself takes around 20 minutes before completing (no error), then essentially any az command is very slow, regardless of whether it's an az ml command or not.

Related command

az ml

Errors

No error, just slow

Issue script & Debug output

az --debug

Command output
cli.knack.cli: Command arguments: ['--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7ff6d84ac7c0>, <function OutputProducer.on_global_arguments at 0x7ff6d81b7060>, <function CLIQuery.on_global_arguments at 0x7ff6d81e85e0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: No module found from index for '['--debug']'
cli.azure.cli.core: Loading all modules and extensions
cli.azure.cli.core: Discovered command modules: ['acr', 'acs', 'advisor', 'ams', 'apim', 'appconfig', 'appservice', 'aro', 'backup', 'batch', 'batchai', 'billing', 'botservice', 'cdn', 'cloud', 'cognitiveservices', 'compute_recommender', 'computefleet', 'config', 'configure', 'consumption', 'container', 'containerapp', 'cosmosdb', 'databoxedge', 'dls', 'dms', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'hdinsight', 'identity', 'interactive', 'iot', 'keyvault', 'lab', 'managedservices', 'maps', 'marketplaceordering', 'monitor', 'mysql', 'netappfiles', 'network', 'policyinsights', 'privatedns', 'profile', 'rdbms', 'redis', 'relay', 'resource', 'role', 'search', 'security', 'servicebus', 'serviceconnector', 'servicefabric', 'signalr', 'sql', 'sqlvm', 'storage', 'synapse', 'util', 'vm']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: acr                       0.327        36       149
cli.azure.cli.core: acs                       0.040        14        77
cli.azure.cli.core: advisor                   0.003         3         6
cli.azure.cli.core: ams                       0.017        22       100
cli.azure.cli.core: apim                      0.012        14        69
cli.azure.cli.core: appconfig                 0.007         9        47
cli.azure.cli.core: appservice                0.046        79       270
cli.azure.cli.core: aro                       0.014         1        10
cli.azure.cli.core: backup                    0.005        16        60
cli.azure.cli.core: batch                     0.223        34       102
cli.azure.cli.core: batchai                   0.008        10        30
cli.azure.cli.core: billing                   0.033        19        52
cli.azure.cli.core: botservice                0.017        12        42
cli.azure.cli.core: cdn                       0.387         8        50
cli.azure.cli.core: cloud                     0.004         1         7
cli.azure.cli.core: cognitiveservices         0.007        10        33
cli.azure.cli.core: compute_recommender       0.004         1         1
cli.azure.cli.core: computefleet              0.013         1         1
cli.azure.cli.core: config                    0.002         2         7
cli.azure.cli.core: configure                 0.002         2         5
cli.azure.cli.core: consumption               0.025         8         9
cli.azure.cli.core: container                 0.022         2        17
cli.azure.cli.core: containerapp              0.235        37       123
cli.azure.cli.core: cosmosdb                  0.038        58       199
cli.azure.cli.core: databoxedge               0.013         5        28
cli.azure.cli.core: dls                       0.004         6        36
cli.azure.cli.core: dms                       0.004         3        22
cli.azure.cli.core: eventgrid                 0.009        25        96
cli.azure.cli.core: eventhubs                 0.020        14        21
cli.azure.cli.core: extension                 0.005         1         7
cli.azure.cli.core: feedback                  0.002         1         2
cli.azure.cli.core: find                      0.003         1         1
cli.azure.cli.core: hdinsight                 0.016         9        42
cli.azure.cli.core: identity                  0.004         2        11
cli.azure.cli.core: interactive               0.001         1         1
cli.azure.cli.core: iot                       0.194        19        82
cli.azure.cli.core: keyvault                  0.021        20       113
cli.azure.cli.core: lab                       0.069         9        29
cli.azure.cli.core: managedservices           0.033         3         8
cli.azure.cli.core: maps                      0.090         5        13
cli.azure.cli.core: marketplaceordering       0.005         1         2
cli.azure.cli.core: monitor                   0.984        17        61
cli.azure.cli.core: mysql                     0.140        16        56
cli.azure.cli.core: netappfiles               0.076         8        17
cli.azure.cli.core: network                   0.337       104       345
cli.azure.cli.core: policyinsights            0.023         9        17
cli.azure.cli.core: privatedns                0.050        14        60
cli.azure.cli.core: profile                   0.010         2         8
cli.azure.cli.core: rdbms                     0.199        51       212
cli.azure.cli.core: redis                     0.004         7        38
cli.azure.cli.core: relay                     0.197         7         8
cli.azure.cli.core: resource                  0.037        52       232
cli.azure.cli.core: role                      0.007        17        61
cli.azure.cli.core: search                    0.015         7        19
cli.azure.cli.core: security                  0.029        47        98
cli.azure.cli.core: servicebus                0.016        14        17
cli.azure.cli.core: serviceconnector          1.002        20       315
cli.azure.cli.core: servicefabric             0.028        27        80
cli.azure.cli.core: signalr                   0.008        10        42
cli.azure.cli.core: sql                       0.035        56       215
cli.azure.cli.core: sqlvm                     0.040         4        20
cli.azure.cli.core: storage                   0.095        59       274
cli.azure.cli.core: synapse                   0.030        54       246
cli.azure.cli.core: util                      0.002         3         7
cli.azure.cli.core: vm                        0.141        58       225
cli.azure.cli.core: Total (65)                5.496      1187      4653
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
cli.azure.cli.core: ml                      159.215        25       165  /home/cecheta/.azure/cliextensions/ml
cli.azure.cli.core: Total (1)               159.215        25       165
cli.azure.cli.core: Loaded 1198 groups, 4818 commands.
cli.azure.cli.core: Updated command index in 0.024 seconds.
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7ff6d7046de0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/cecheta/.azure/commands/2025-02-09.19-39-15.unknown_command.38325.log'.
az_command_data_logger: command args: --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7ff6d70a2d40>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7ff6d70a2de0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7ff6d70a2f20>, <function register_upcoming_breaking_change_info.<locals>.update_breaking_change_info at 0x7ff6d70a2fc0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []

     /\
    /  \    _____   _ _  ___ _
   / /\ \  |_  / | | | \'__/ _\
  / ____ \  / /| |_| | | |  __/
 /_/    \_\/___|\__,_|_|  \___|


Welcome to the cool new Azure CLI!

Use `az --version` to display the current version.
Here are the base commands:

    account             : Manage Azure subscription information.
    acr                 : Manage private registries with Azure Container Registries.
    ad                  : Manage Microsoft Entra ID (formerly known as Azure Active Directory, Azure
                         AD, AAD) entities needed for Azure role-based access control (Azure RBAC)
                         through Microsoft Graph API.
    advisor             : Manage Azure Advisor.
    afd                 : Manage Azure Front Door Standard/Premium.
    aks                 : Manage Azure Kubernetes Services.
    ams                 : Manage Azure Media Services resources.
    apim                : Manage Azure API Management services.
    appconfig           : Manage App Configurations.
    appservice          : Manage App Service plans.
    aro                 : Manage Azure Red Hat OpenShift clusters.
    backup              : Manage Azure Backups.
    batch               : Manage Azure Batch.
    bicep               : Bicep CLI command group.
    billing             : Manage Azure Billing.
    bot                 : Manage Microsoft Azure Bot Service.
    cache               : Commands to manage CLI objects cached using the `--defer` argument.
    capacity            : Manage capacity.
    cdn                 : Manage Azure Content Delivery Networks (CDNs).
    cloud               : Manage registered Azure clouds.
    cognitiveservices   : Manage Azure Cognitive Services accounts.
    compute-fleet       : Manage for Azure Compute Fleet.
    compute-recommender : Manage sku/zone/region recommender info for compute resources.
    config              : Manage Azure CLI configuration.
    configure           : Manage Azure CLI configuration. This command is interactive.
    connection          : Commands to manage Service Connector local connections which allow local
                         environment to connect Azure Resource. If you want to manage connection for
                         compute service, please run 'az webapp/containerapp/spring connection'.
    consumption         : Manage consumption of Azure resources.
    container           : Manage Azure Container Instances.
    containerapp        : Manage Azure Container Apps.
    cosmosdb            : Manage Azure Cosmos DB database accounts.
    data-boundary       : Data boundary operations.
    databoxedge         : Manage device with databoxedge.
    deployment          : Manage Azure Resource Manager template deployment at subscription scope.
    deployment-scripts  : Manage deployment scripts at subscription or resource group scope.
    disk                : Manage Azure Managed Disks.
    disk-access         : Manage disk access resources.
    disk-encryption-set : Disk Encryption Set resource.
    dls                 : Manage Data Lake Store accounts and filesystems.
    dms                 : Manage Azure Data Migration Service (classic) instances.
    eventgrid           : Manage Azure Event Grid topics, domains, domain topics, system topics
                         partner topics, event subscriptions, system topic event subscriptions and
                         partner topic event subscriptions.
    eventhubs           : Eventhubs.
    extension           : Manage and update CLI extensions.
    feature             : Manage resource provider features.
    feedback            : Send feedback to the Azure CLI Team.
    find                : I'm an AI robot, my advice is based on our Azure documentation as well as
                         the usage patterns of Azure CLI and Azure ARM users. Using me improves
                         Azure products and documentation.
    functionapp         : Manage function apps. To install the Azure Functions Core tools see
                         https://github.com/Azure/azure-functions-core-tools.
    group               : Manage resource groups and template deployments.
    hdinsight           : Manage HDInsight resources.
    identity            : Managed Identities.
    image               : Manage custom virtual machine images.
    interactive         : Start interactive mode. Installs the Interactive extension if not
                         installed already.
    iot                 : Manage Internet of Things (IoT) assets.
    keyvault            : Manage KeyVault keys, secrets, and certificates.
    lab                 : Manage azure devtest labs.
    lock                : Manage Azure locks.
    logicapp            : Manage logic apps.
    login               : Log in to Azure.
    logout              : Log out to remove access to Azure subscriptions.
    managed-cassandra   : Azure Managed Cassandra.
    managedapp          : Manage template solutions provided and maintained by Independent Software
                         Vendors (ISVs).
    managedservices     : Manage the registration assignments and definitions in Azure.
    maps                : Manage Azure Maps.
    mariadb             : Manage Azure Database for MariaDB servers.
    ml                  : Manage Azure Machine Learning resources with the Azure CLI ML extension
                         v2.
    monitor             : Manage the Azure Monitor Service.
    mysql               : Manage Azure Database for MySQL servers.
    netappfiles         : Manage Azure NetApp Files (ANF) Resources.
    network             : Manage Azure Network resources.
    policy              : Manage resource policies.
    postgres            : Manage Azure Database for PostgreSQL servers.
    ppg                 : Manage Proximity Placement Groups.
    private-link        : Private-link association CLI command group.
    provider            : Manage resource providers.
    redis               : Manage dedicated Redis caches for your Azure applications.
    relay               : Manage Azure Relay Service namespaces, WCF relays, hybrid connections, and
                         rules.
    resource            : Manage Azure resources.
    resourcemanagement  : Resourcemanagement CLI command group.
    rest                : Invoke a custom request.
    restore-point       : Manage restore point with res.
    role                : Manage Azure role-based access control (Azure RBAC).
    search              : Manage Azure Search services, admin keys and query keys.
    security            : Manage your security posture with Microsoft Defender for Cloud.
    servicebus          : Servicebus.
    sf                  : Manage and administer Azure Service Fabric clusters.
    sig                 : Manage shared image gallery.
    signalr             : Manage Azure SignalR Service.
    snapshot            : Manage point-in-time copies of managed disks, native blobs, or other
                         snapshots.
    sql                 : Manage Azure SQL Databases and Data Warehouses.
    sshkey              : Manage ssh public key with vm.
    stack               : A deployment stack is a native Azure resource type that enables you to
                         perform operations on a resource collection as an atomic unit.
    staticwebapp        : Manage static apps.
    storage             : Manage Azure Cloud Storage resources.
    survey              : Take Azure CLI survey.
    synapse             : Manage and operate Synapse Workspace, Spark Pool, SQL Pool.
    tag                 : Tag Management on a resource.
    term                : Manage marketplace agreement with marketplaceordering.
    ts                  : Manage template specs at subscription or resource group scope.
    upgrade             : Upgrade Azure CLI and extensions.
    version             : Show the versions of Azure CLI modules and extensions in JSON format by
                         default or format configured by --output.
    vm                  : Manage Linux or Windows virtual machines.
    vmss                : Manage groupings of virtual machines in an Azure Virtual Machine Scale Set
                         (VMSS).
    webapp              : Manage web apps.
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7ff6d7047060>]
az_command_data_logger: exit code: 0
cli.__main__: Command ran in 165.750 seconds (init: 0.565, invoke: 165.185)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3712 in cache file under /home/cecheta/.azure/telemetry/20250209193915755
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.68.0/libexec/bin/python /home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.68.0/libexec/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/cecheta/.azure /home/cecheta/.azure/telemetry/20250209193915755"
telemetry.process: Return from creating process 38723
telemetry.main: Finish creating telemetry upload process.

Expected behavior

It shoudn't take so long to run az.

Environment Summary

azure-cli 2.68.0

core 2.68.0
telemetry 1.1.0

Extensions:
ml 2.34.0

Dependencies:
msal 1.31.1
azure-mgmt-resource 23.1.1

Python location '/home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.68.0/libexec/bin/python'
Extensions directory '/home/cecheta/.azure/cliextensions'

Python (Linux) 3.12.9 (main, Feb 4 2025, 14:38:38) [GCC 11.4.0]

I am running on WSL Ubuntu

Additional context

From the logs above, it takes 159 seconds to load the ml extension.

@cecheta cecheta added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Feb 9, 2025
@yonzhan
Copy link
Collaborator

yonzhan commented Feb 9, 2025

Thank you for opening this issue, we will look into it.

Copy link

github-actions bot commented Feb 9, 2025

Here are some similar issues that might help you. Please check if they can solve your problem.


Possible solution (Extracted from existing issue, might be incorrect; please verify carefully)

As a workaround, download from GitHub is much faster: https://github.com/Azure/azure-cli/releases

Reference:

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AzureAppServiceCLI, @antcp.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @adamedx.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AzureAppServiceCLI, @antcp.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @adamedx.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AzureAppServiceCLI, @antcp.

@cecheta
Copy link
Member Author

cecheta commented Feb 26, 2025

I have found that this issue is a duplicate of #27486, therefore closing.

The cause seems to be Docker Desktop with WSL2. It creates a symlink on WSL from ~/.azure to the Windows filesystem, which is then very slow when the Azure CLI tries to install and use the ml extension from the ~/.azure directory (on Windows).

The fix was to remove the symlink, then install the ml extension again.

@cecheta cecheta closed this as completed Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. Machine Learning az ml Possible-Solution Service Attention This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

4 participants