Issue with Monitor DataCollectionRule / API Version 2022-06-01: expose detail for error

[teowa]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Service Used

Monitor

API Versions Used

2022-06-01

Description

Error details should be exposed, PR #482 only handles the case for long-running-operation.

Existing error looks like

 Error: updating Data Collection Rule (Subscription: "<redacted>"
│ Resource Group Name: "wt-example-resources"
│ Data Collection Rule Name: "example-rule"): unexpected status 400 (400 Bad Request) with error: InvalidPayload: Data collection rule is invalid
│
│   with azurerm_monitor_data_collection_rule.example,
│   on main.tf line 120, in resource "azurerm_monitor_data_collection_rule" "example":
│  120: resource "azurerm_monitor_data_collection_rule" "example" {
│
│ updating Data Collection Rule (Subscription: "49824ffe-67ce-4765-be73-93c2ac1b55a8"
│ Resource Group Name: "wt-example-resources"
│ Data Collection Rule Name: "example-rule"): unexpected status 400 (400 Bad Request) with error: InvalidPayload: Data collection rule is invalid
╵

repro config

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=4.7.0"
    }
  }
}

provider "azurerm" {
  features {}
}
resource "azurerm_resource_group" "example" {
  name     = "wt-example-resources"
  location = "southeastasia"
}
resource "azurerm_log_analytics_workspace" "example" {
  name                = "examplewt-workspace"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
}
resource "azurerm_monitor_data_collection_rule" "example" {
  name                = "example-rule"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  #data_collection_endpoint_id = azurerm_monitor_data_collection_endpoint.example.id

  destinations {
    log_analytics {
      workspace_resource_id = azurerm_log_analytics_workspace.example.id
      name                  = "example-destination-log"
    }
  }
  data_flow {
    streams      = ["Microsoft-InsightsMetrics", "Microsoft-Syslog", "Microsoft-Perf"]
    destinations = ["example-destination-log"]
  }
  data_flow {
    streams       = ["Custom-MyTableRawData"]
    destinations  = ["example-destination-log"]
    output_stream = "Microsoft-Syslog"
    transform_kql = "source | project TimeGenerated = Time, Computer, Message = AdditionalContext,"
  }
  data_sources {
    syslog {
      facility_names = ["*"]
      log_levels     = ["*"]
      name           = "example-datasource-syslog"
      streams        = ["Microsoft-Syslog"]
    }
  }
  stream_declaration {
    stream_name = "Custom-MyTableRawData"
    column {
      name = "Time"
      type = "datetime"
    }
    column {
      name = "Computer"
      type = "string"
    }
    column {
      name = "AdditionalContext"
      type = "string"
    }
  }
}

API error has more details:

{"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"InvalidTransformQuery","message":"Error occurred while compiling query in query: SyntaxError:0x00000001 at 1:76 : extraneous input ',' expecting <EOF>","target":"properties.dataFlows[1]"}]}}

az rest --method put --url '/subscriptions/<redacted>/resourceGroups/wt-example-resources/providers/Microsoft.Insights/dataCollectionRules/example-rule?api-version=2022-06-01' --body '{"properties":{"streamDeclarations":{"Custom-MyTableRawData":{"columns":[{"name":"Time","type":"datetime"},{"name":"Computer","type":"string"},{"name":"AdditionalContext","type":"string"}]}},"dataSources":{"syslog":[{"streams":["Microsoft-Syslog"],"facilityNames":["*"],"logLevels":["*"],"name":"example-datasource-syslog"}]},"destinations":{"logAnalytics":[{"workspaceResourceId":"/subscriptions/<redacted>/resourceGroups/wt-example-resources/providers/Microsoft.OperationalInsights/workspaces/examplewt-workspace","workspaceId":"64b135a2-b17d-46bd-b047-38314961cd93","name":"example-destination-log"}]},"dataFlows":[{"streams":["Microsoft-InsightsMetrics","Microsoft-Syslog","Microsoft-Perf"],"destinations":["example-destination-log"]},{"streams":["Custom-MyTableRawData"],"destinations":["example-destination-log"],"transformKql":"source | project TimeGenerated = Time, Computer, Message = AdditionalContext,","outputStream":"Microsoft-Syslog"}]},"location":"southeastasia"}' --debug
...
cli.azure.cli.core.util: Request URL: 'https://management.azure.com/subscriptions/<redacted>/resourceGroups/wt-example-resources/providers/Microsoft.Insights/dataCollectionRules/example-rule?api-version=2022-06-01'
cli.azure.cli.core.util: Request method: 'PUT'
cli.azure.cli.core.util: Request headers:
cli.azure.cli.core.util:     'User-Agent': 'python/3.11.9 (Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35) AZURECLI/2.63.0 (HOMEBREW)'
cli.azure.cli.core.util:     'Accept-Encoding': 'gzip, deflate'
cli.azure.cli.core.util:     'Accept': '*/*'
cli.azure.cli.core.util:     'Connection': 'keep-alive'
cli.azure.cli.core.util:     'x-ms-client-request-id': '12f97318-4769-4404-bb51-5700f63beb12'
cli.azure.cli.core.util:     'Content-Type': 'application/json'
cli.azure.cli.core.util:     'CommandName': 'rest'
cli.azure.cli.core.util:     'ParameterSetName': '--method --url --body --debug'
cli.azure.cli.core.util:     'Authorization': 'Bearer eyJ0eXAiOiJKV...'
cli.azure.cli.core.util:     'Content-Length': '1054'
cli.azure.cli.core.util: Request body:
cli.azure.cli.core.util: {"properties": {"streamDeclarations": {"Custom-MyTableRawData": {"columns": [{"name": "Time", "type": "datetime"}, {"name": "Computer", "type": "string"}, {"name": "AdditionalContext", "type": "string"}]}}, "dataSources": {"syslog": [{"streams": ["Microsoft-Syslog"], "facilityNames": ["*"], "logLevels": ["*"], "name": "example-datasource-syslog"}]}, "destinations": {"logAnalytics": [{"workspaceResourceId": "/subscriptions/<redacted>/resourceGroups/wt-example-resources/providers/Microsoft.OperationalInsights/workspaces/examplewt-workspace", "workspaceId": "64b135a2-b17d-46bd-b047-38314961cd93", "name": "example-destination-log"}]}, "dataFlows": [{"streams": ["Microsoft-InsightsMetrics", "Microsoft-Syslog", "Microsoft-Perf"], "destinations": ["example-destination-log"]}, {"streams": ["Custom-MyTableRawData"], "destinations": ["example-destination-log"], "transformKql": "source | project TimeGenerated = Time, Computer, Message = AdditionalContext,", "outputStream": "Microsoft-Syslog"}]}, "location": "southeastasia"}
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/<redacted>/resourceGroups/wt-example-resources/providers/Microsoft.Insights/dataCollectionRules/example-rule?api-version=2022-06-01 HTTP/1.1" 400 288
cli.azure.cli.core.util: Response status: 400
cli.azure.cli.core.util: Response headers:
cli.azure.cli.core.util:     'Cache-Control': 'no-cache'
cli.azure.cli.core.util:     'Pragma': 'no-cache'
cli.azure.cli.core.util:     'Content-Length': '288'
cli.azure.cli.core.util:     'Content-Type': 'application/json'
cli.azure.cli.core.util:     'Expires': '-1'
cli.azure.cli.core.util:     'x-ms-ratelimit-remaining-subscription-resource-requests': '149'
cli.azure.cli.core.util:     'Request-Context': 'appId=cid-v1:2bbfbac8-e1b0-44af-b9c6-3a40669d37e3'
cli.azure.cli.core.util:     'x-ms-correlation-request-id': '6b33863a-c24b-434c-8673-f78a3d04983d'
cli.azure.cli.core.util:     'x-ms-client-request-id': '12f97318-4769-4404-bb51-5700f63beb12'
cli.azure.cli.core.util:     'x-ms-routing-request-id': 'JAPANEAST:20241031T075142Z:6b33863a-c24b-434c-8673-f78a3d04983d'
cli.azure.cli.core.util:     'x-ms-request-id': 'e791cf4e-9f0a-4c40-b420-73d4235e7e70'
cli.azure.cli.core.util:     'api-supported-versions': '2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11, 2024-03-11'
cli.azure.cli.core.util:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.util:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.util:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.util:     'X-MSEdge-Ref': 'Ref A: B825BFEAC29A47D9B26385BDD78FF422 Ref B: TYO201100114011 Ref C: 2024-10-31T07:51:42Z'
cli.azure.cli.core.util:     'Date': 'Thu, 31 Oct 2024 07:51:44 GMT'
cli.azure.cli.core.util: Response content:
cli.azure.cli.core.util: {"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"InvalidTransformQuery","message":"Error occurred while compiling query in query: SyntaxError:0x00000001 at 1:76 : extraneous input ',' expecting <EOF>","target":"properties.dataFlows[1]"}]}}

swagger has defined the error struct:
https://github.com/Azure/azure-rest-api-specs/blob/bbf2878dfeff8ee6f5a8ac6efad61c801972cfb5/specification/common-types/resource-management/v2/types.json#L285-L337

https://github.com/Azure/azure-rest-api-specs/blob/065bac6153f6a83de668f55803270bc6f31f5bc3/specification/monitor/resource-manager/Microsoft.Insights/stable/2022-06-01/dataCollectionRules_API.json#L200-L206

Maybe we should consume the swagger and generate the CustomErrorParser, but I am not sure.

References

hashicorp/terraform-provider-azurerm#27820