fix(ml): populate subscriptionId and resourceGroup for CLI-created datastores

[fmabroukmsft]

Problem

When creating Azure storage datastores via az ml datastore create, the resulting datastore is missing subscriptionId and resourceGroup in its properties. This causes downstream operations like sharing data assets to registries to fail with a 400 error.

UI-created datastores correctly have these fields populated. Only CLI-created datastores are affected.

Before (CLI-created datastore)

{
  "subscriptionId": null,
  "resourceGroup": null,
  "datastoreType": "AzureBlob",
  "accountName": "mystorageaccount",
  "containerName": "mycontainer"
}

After (with this fix)

{
  "subscriptionId": "8f338f6e-...",
  "resourceGroup": "my-resource-group",
  "datastoreType": "AzureBlob",
  "accountName": "mystorageaccount",
  "containerName": "mycontainer"
}

Root Cause

The SDK entity classes (AzureBlobDatastore, AzureFileDatastore, AzureDataLakeGen2Datastore) do not include subscription_id or resource_group in their _to_rest_object() serialization, even though the underlying REST models (RestAzureBlobDatastore etc.) accept these optional fields and the service stores them correctly when provided.

Fix

This PR adds a helper _create_or_update_with_arm_scope() in the CLI extension's datastore command handler that:

1. Builds the REST object via the SDK's _to_rest_object()
2. Injects the workspace's subscriptionId and resourceGroup into the REST body when the entity doesn't provide them
3. Calls the service directly

This is a targeted CLI-extension-level fix. A more comprehensive fix should be made in the azure-ai-ml SDK to add these fields to the YAML schema, entity constructors, and _to_rest_object() methods.

Affected Commands

• az ml datastore create
• az ml datastore update

Testing

• ✅ Manually tested: created a blob datastore via CLI, verified subscriptionId and resourceGroup are now populated in the ARM API response
• ✅ Verified existing datastores are unaffected
• ✅ Verified the fix applies to all three Azure storage types (Blob, File, ADLS Gen2)

Related

• ICM 716428613 — customer-reported issue with malformed CLI-created datastores

[azure-client-tools-bot-prd]

Validation for Breaking Change Starting...

Thanks for your contribution!

[azure-client-tools-bot-prd]

Hi @fmabroukmsft,
Please write the description of changes which can be perceived by customers into HISTORY.rst.
If you want to release a new extension version, please update the version in setup.py as well.

[yonzhan]

Thank you for your contribution! We will review the pull request and get back to you soon.

[github-actions]

The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR.

Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions).
After that please run the following commands to enable git hooks:

pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>

[github-actions]

Hi @fmabroukmsft

Release Suggestions

Module: machinelearningservices

• Please log updates into to src/machinelearningservices/HISTORY.rst
• Update VERSION to 2.42.1 in src/machinelearningservices/setup.py

Notes

• For more info about extension versioning, please refer to Extension version schema

[Copilot]

Pull request overview

This PR addresses an Azure ML CLI parity issue where datastores created/updated via az ml datastore create/update are missing ARM-scope fields (subscriptionId, resourceGroup), causing downstream operations (e.g., sharing to registries) to fail.

Changes:

• Added _create_or_update_with_arm_scope() to serialize a datastore to a REST object, backfill subscriptionId/resourceGroup for Azure storage datastores, and call the service.
• Routed ml_datastore_create and ml_datastore_update through the new helper.

[Copilot]

This adds a dependency on azure.ai.ml.entities._datastore.azure_storage which is a private SDK module (leading underscore). That makes the CLI extension fragile across azure-ai-ml upgrades. Consider detecting Azure storage datastores via public surface (e.g., the datastore entity type string like azure_blob / azure_file / azure_data_lake_gen2) instead of importing private classes.

[Copilot]

This helper relies on private MLClient internals (_operation_scope._resource_group_name, datastores._workspace_name, and datastores._operation). Since ml_datastore_create/update already receive resource_group_name and workspace_name, it would be more robust to pass those into this helper (or use the public ml_client.datastores.create_or_update path) to avoid breaking when SDK internals change.

Suggested change

	"""Create or update a datastore, backfilling subscription and resource group.
	The SDK's ``_to_rest_object`` does not populate ``subscriptionId`` and
	``resourceGroup`` in the request body for Azure storage datastores. When
	these fields are missing the created datastore lacks ARM scope, which
	breaks downstream operations such as sharing data assets to a registry.
	This helper builds the REST object, injects the workspace's subscription
	and resource group when the datastore entity does not carry them, and
	then calls the service directly.
	"""
	ds_request = datastore._to_rest_object() # pylint: disable=protected-access
	if isinstance(datastore, _AZURE_STORAGE_DATASTORE_TYPES):
	subscription_id = ml_client._operation_scope.subscription_id # pylint: disable=protected-access
	resource_group = ml_client._operation_scope._resource_group_name # pylint: disable=protected-access
	props = ds_request.properties
	if props is not None:
	if not getattr(props, 'subscription_id', None):
	props.subscription_id = subscription_id
	if not getattr(props, 'resource_group', None):
	props.resource_group = resource_group
	datastore_resource = ml_client.datastores._operation.create_or_update( # pylint: disable=protected-access
	name=datastore.name,
	resource_group_name=ml_client._operation_scope._resource_group_name, # pylint: disable=protected-access
	workspace_name=ml_client.datastores._workspace_name, # pylint: disable=protected-access
	body=ds_request,
	skip_validation=True,
	)
	return Datastore._from_rest_object(datastore_resource) # pylint: disable=protected-access
	"""Create or update a datastore using the public datastores client.
	This helper avoids relying on private MLClient internals by delegating to
	``ml_client.datastores.create_or_update``.
	"""
	return ml_client.datastores.create_or_update(datastore)

[Copilot]

skip_validation=True is a behavior change from the previous implementation (ml_client.datastores.create_or_update(datastore) did not force this flag). If skip_validation disables service-side validation, this could allow invalid datastore definitions or produce less actionable errors. Consider preserving the prior default (omit the flag / set it explicitly to the previous default) or plumb it as an optional parameter so existing behavior is unchanged unless needed.

Suggested change

skip_validation=True,

[Copilot]

This change affects the HTTP request body for az ml datastore create/update (injecting subscriptionId/resourceGroup). There are existing datastore scenario tests with recordings, but none assert the new fields. Please update/add a scenario test assertion that the created datastore response includes subscription_id and resource_group (or the expected keys in CLI output), and refresh the relevant recordings if playback matching depends on request bodies.

[Copilot]

Same as create: this alters the request body for az ml datastore update by injecting ARM scope fields. Please ensure scenario coverage includes an update case that verifies the updated datastore has subscription_id and resource_group populated, and refresh recordings if needed.

[fmabroukmsft]

Companion SDK fix submitted: Azure/azure-sdk-for-python#46067

The SDK fix adds subscription_id and resource_group natively to the entity classes, YAML schema, and round-trip serialization. Once the SDK fix ships, the workaround in this CLI PR can be simplified to just call ml_client.datastores.create_or_update() directly (the current behavior) — but the CLI-level backfill will still serve as defense-in-depth for users on older SDK versions.