Reservation recommendations ingestion fails (MCA) on v0.7

[kasimrehman]

🐛 Problem

Errors are seen in Data Factory relating to the ingestion of reservation recommendations:

ErrorCode=UserErrorInvalidDataValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'Net Savings' contains an invalid value '{"currency":"USD","value":114961.468965322154294}'. Cannot convert '{"currency":"USD","value":114961.468965322154294}' to type 'Decimal'.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.FormatException,Message=Input string was not in a correct format.,Source=mscorlib,'

ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'SKU' specified in column mapping cannot be found in 'billingProfiles/XXXX-XXXX-XXX-XXX/PRE-ReservationsRecommendations-Kits/202502121437/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/part0.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'

Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'finopshubenybzcya46q36'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/02/providers/microsoft.billing/billingaccounts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_2019-05-31/billingprofiles/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '66c1d54e-301f-008d-496f-7cf906000000'. TimeStamp: 'Tue, 11 Feb 2025 10:26:38 GMT'.

👣 Repro steps

TODO: Add repro steps below:
1.Create a data export from Azure Cost Management for reservation recommendations (MCA)
2. Wait for ingestion
3. Look at the monitor in Data Factory and examine the errors

🤔 Expected

Ingestion should happen without errors

[flanakin]

Thanks for reporting. At first glance, this looks like exports changed their format. Can you report this via a support request back to Cost Management? The export schema should not change for a specific version. They should revert their change.

For reference, here's an example dataset that we expect. You can see the NetSavings column is decimal and not a JSON object.
https://github.com/microsoft/finops-toolkit/releases/latest/download/dataset-examples.zip

In the meantime, we can update our logic to handle both scenarios, but the correct fix is for them to revert their breaking change.

[darushah]

@flanakin a quick question for the second error "ErrorCode=MappingColumnNameNotFoundInSourceFile" for the column 'SKU'.

For MCA as per this doc https://learn.microsoft.com/en-us/azure/cost-management-billing/dataset-schema/reservation-recommendations-mca RI recommendation does not have any column by the name "SKU". Instead for MCA it is "SkuName".

Hence, will FinOps toolkit fix the mapping of this column between EA and MCA RI recommendation dataset export data.

[kasimrehman]

Similarly @flanakin, It seems there are two schemas for reservation recommendations, at least in the REST API: Legacy and Modern, as specified here. The modern format has the net saving as an amount JSON object whereas the legacy one has it as a number.