Dashboard properties missing in Pulumi state

[mikocot]

What happened?

We've deployed an empty dashboard object with azure native provider. That dash has empty lens collection.
We've then added numerous dashboard lenses separately and set ignoreChanges property for Pulumi to not clear those parts during the nest pulumi up.

In fact this last part wasn't even needed until we started using drift detection pulumi refresh in our CI/CD. However, once the refresh ran on the stack, pulumi became aware of the new dashboards content and tried to clean it. It's all as expected until now.

Unfortunately, the data pulled into the state is not complete and now seems to cause validation issue of the dashboard object, even though it's perfectly functional in Azure. When we pull dashboard structure from the portal or CLI, the missing properties are there.

Example

Code for the empty dashboard:

  Dashboard = new AzurePortal.Dashboard($"{name}-",
      new AzurePortal.DashboardArgs
      {
          ResourceGroupName = args.ResourceGroup.Name,
          Location = args.ResourceGroup.Location,
          Lenses = new[]
          {
              new AzurePortal.Inputs.DashboardLensArgs
              {
                  Order = 0,
                  Parts = Array.Empty<AzurePortal.Inputs.DashboardPartsArgs>()
              }
          }
      }, new CustomResourceOptions { IgnoreChanges = { "lenses" } });

error from pulumi up

Diagnostics:
azure-native:portal:Dashboard (dashboard-beacon-platform-pr753-eastus2-)
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[0].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[1].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[2].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[3].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[4].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[5].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[6].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[7].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[8].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[9].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[10].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[11].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[12].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[13].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[14].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[15].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[16].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[17].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[18].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[19].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[20].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[21].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[22].metadata.type'
error: azure-native:portal/v20200901preview:Dashboard resource 'dashboard-beacon-platform-pr753-eastus2-' has a problem: missing required property 'lenses[0].parts[23].metadata.type'

Dashboard lenses in pulumi state (as presented by app.pulumi.com)

[
  {
    "order": 0,
    "parts": [
      {
        "metadata": {},
        "position": {
          "colSpan": 5,
          "rowSpan": 3,
          "x": 0,
          "y": 0
        }
      },
      {
        "metadata": {},
        "position": {
          "colSpan": 5,
          "rowSpan": 3,
          "x": 5,
          "y": 0
        }
      }
(.......)
]

What CLI (az portal dashboard show) return reports for that same dashboards - similar result if browsing the template in the portal, or exporting dashboard:

"lenses": {
    "0": {
      "metadata": null,
      "order": 0,
      "parts": {
        "0": {
          "metadata": {
            "filters": {
              "cloud/roleInstance": {
                "model": {
                  "operator": "startsWith",
                  "substring": "app-ingestion-ca38056ed6"
                }
              }
            },
            "inputs": [
              {
                "isOptional": true,
                "name": "options",
                "value": {
                  "chart": {
                    "filterCollection": {
                      "filters": [
                        {
                          "key": "cloud/roleInstance",
                          "operator": 3,
                          "values": [
                            "app-ingestion-ca38056ed6"
                          ]
                        }
                      ]
                    },
                    "metrics": [
                      {
                        "aggregationType": 4,
                        "metricVisualization": {
                          "displayName": "Server request rate"
                        },
                        "name": "requests/rate",
                        "namespace": "microsoft.insights/components",
                        "resourceMetadata": {
                          "id": "/subscriptions/[REDACTED]",
                          "resourceGroup": "[REDACTED]"
                        }
                      }
                    ],
                    "title": "Ingestion - avg request rate",
                    "titleKind": 2,
                    "visualization": {
                      "axisVisualization": {
                        "x": {
                          "axisType": 2,
                          "isVisible": true
                        },
                        "y": {
                          "axisType": 1,
                          "isVisible": true
                        }
                      },
                      "chartType": 2,
                      "legendVisualization": {
                        "hideSubtitle": true,
                        "isVisible": true,
                        "position": 2
                      }
                    }
                  }
                }
              },
              {
                "isOptional": true,
                "name": "sharedTimeRange"
              }
            ],
            "settings": {
              "content": {
                "options": {
                  "chart": {
                    "disablePinning": true,
                    "metrics": [
                      {
                        "aggregationType": 4,
                        "metricVisualization": {
                          "displayName": "Server request rate"
                        },
                        "name": "requests/rate",
                        "namespace": "microsoft.insights/components",
                        "resourceMetadata": {
                          "id": "[REDACTED]",
                          "resourceGroup": "[REDACTED]"
                        }
                      }
                    ],
                    "title": "Ingestion - avg request rate",
                    "titleKind": 2,
                    "visualization": {
                      "axisVisualization": {
                        "x": {
                          "axisType": 2,
                          "isVisible": true
                        },
                        "y": {
                          "axisType": 1,
                          "isVisible": true
                        }
                      },
                      "chartType": 2,
                      "legendVisualization": {
                        "hideSubtitle": true,
                        "isVisible": true,
                        "position": 2
                      }
                    }
                  }
                }
              }
            },
            "type": "Extension/HubsExtension/PartType/MonitorChartPart"
          },
          "position": {
            "colSpan": 5,
            "metadata": null,
            "rowSpan": 3,
            "x": 0,
            "y": 0
          }
        },
       (......)
   }

Output of pulumi about

we use the latest pulumi GH actions for deployment, so printing my personal pulumi about isn't relevant

Additional context

I believe there are more layers of this issue - another one, a reason why we were forced to use a deployment object to deploy dashboard parts is explained by this issue. But let's focus on this issue for now. Especially since I believe with dashboards it's a common scenario that they are designed in the portal, manually and not fully managed within pulumi. Yet, there needs to be a way that pulumi does not destroy it or break deployments altogether.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mjeffryes]

Thanks for reporting this @mikocot. Given that there's richer information returned in the CLI, we might be able to improve the state import for this resource.

[mikocot]

Thanks, we've now created also this issue that is likely related: #3023

The problem being that the types on pulumi side don't seem to reflect parameters in Azure's API. For us, fixing any of those two should be sufficient. But its not unlikely that fixing one will fix the other as well.

[mjeffryes]

based on further investigation on #3023, it looks like we're missing some types in the upstream azure spec.

[mikocot]

Hm, AFAIR e.g. the inputs property seems to exist on Pulumi side and mastching the Azure one, but it's still not mapped. Why?

Also I do not know if type is the only missing field, but if that's enough to pass the validation, mapping it would be already a step forward (as we can ignore the changes for now) and deploy them using separate deployment object.

Now, the third aspect of it is - why do we even get this exception if we set to ignore ghe whole lenses property?

[mjeffryes]

Hm, AFAIR e.g. the inputs property seems to exist on Pulumi side and mastching the Azure one, but it's still not mapped. Why?

inputs looks like it's a subfield of parts.metadata, the whole definition of metadata is wrong, so I don't think any of it maps out to state.

Also I do not know if type is the only missing field, but if that's enough to pass the validation, mapping it would be already a step forward (as we can ignore the changes for now) and deploy them using separate deployment object.

The parts field can contain several different types of tiles, so there's several types to map here. I filed a ticket upstream (Azure/azure-rest-api-specs#27465); we'd definitely prefer to get this fixed in the schema rather than try to reverse engineer it.

Now, the third aspect of it is - why do we even get this exception if we set to ignore ghe whole lenses property?

Based on the error, it looks like the lenses property may have already been added to the stack state before you added it to ignoreChanges? I wonder if we could avoid saving any state for lenses by starting out with IgnoreChanges = { "lenses" } when doing the initial dashboard create. That might let us side step the problem for now.

[mikocot]

Thanks,

Based on the error, it looks like the lenses property may have already been added to the stack state before you added it to ignoreChanges? I wonder if we could avoid saving any state for lenses by starting out with IgnoreChanges = { "lenses" } when doing the initial dashboard create. That might let us side step the problem for now.

We specify lenses as in the code provided above. We don't add any other content to lenses that would be registered by pulumi state (it's done via deployment which does not seem to update pulumi state for other resources than deployment itself). The reason lenses have any content in it is the refresh that we run in the pipeline, that fills it with partial data, and which AFAIR does not respect ignoreChange, and which sounds reasonable.

Not sure if I understand what else can we do then? Not specify lenses property at all in the initial definition? I believe this is the least we had to put there to have it deployed, but we can retry,

As for the inputs field, you're probably right. I didn't compare the types directly, just saw it exists on both sides.
Now, as I've mentioned on slack, I believe the easiest way to maintain dashboard content might be to become flexible about what is their content and allow lenses as JSON instead (anything serializable to json), instead of maintaining strongly types param classes and making sure it follows any changes on the azure side - if they even decide to share an official schema.

Similarly to what happens in Logic App workflow, where you end up with:

//
// Summary:
//     The definition.
[Input("definition", false, false)]
public Input<object>? Definition { get; set; }

Surely, it's nicer to declare a type having complete strongly typed structure and intellisense, but on the other hand everytime there is an unmapped changed you're blocked. So in the long term it might not be worth it. And the underlying structure can get pretty complex.

[mjeffryes]

Ah, yes, the refresh would pull in the incomplete state regardless of the ignoreChanges property. Treating the lenses as JSON is a good idea, but not something we already support in azure-native. (We've had a related tracking issue for this for a while #68)

Any chance the azure classic resource an option for you in this scenario? It looks like that resource is defined with a dashboardProperties field that's just JSON as you were suggesting.

[mjeffryes]

I've restated #3023 around adding support for handling Dashboard parts as untyped JSON. While this is the older issue and has more discussion, I think the problem is stated a little more clearly there, so I'm going to dupe this ticket against #3023 and we'll continue work over there.