diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index f41313eb53..8735158c99 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -24,6 +24,8 @@ If you're unfamiliar with Infrastructure as Code, or wonder how you can use the - [Bicep to ARM conversion script](./UtilitiesConversionScript) - [Pipelines](./Pipelines) - [Design](./PipelinesDesign) + - [Parameter File Tokens](./ParameterFileTokens) + - [Removal action](./PipelineRemovalAction) - [Usage](./PipelinesUsage) - [Contribution Guide](./ContributionGuide) - [Known Issues](./KnownIssues) diff --git a/docs/wiki/ModulesDesign.md b/docs/wiki/ModulesDesign.md index 76671eb299..865c006da3 100644 --- a/docs/wiki/ModulesDesign.md +++ b/docs/wiki/ModulesDesign.md @@ -499,9 +499,6 @@ While exceptions might be needed, the following guidance should be followed as m > name: '${deployment().name}-Table-${index}' > ``` - - - ## Outputs - Output names are in camelCase, i.e `storageAccountResourceId` diff --git a/docs/wiki/PipelineRemovalAction.md b/docs/wiki/PipelineRemovalAction.md new file mode 100644 index 0000000000..4233b488c1 --- /dev/null +++ b/docs/wiki/PipelineRemovalAction.md @@ -0,0 +1,70 @@ +# Removal action + +This section describes how the removal of resources deployed by a module is performed and how to modify the default behaviour if a specific module or resource type needs it. + +--- + +### _Navigation_ + +- [Overview](#Overview) +- [How it works](#how-it-works) +- [Create a specialized removal procedure](#create-a-specialized-removal-procedure) + +--- + +# Overview + +The Removal action is triggered after the deployment completes. This is used for several reasons: +- Make sure to keep the validation subscription cost as low as possible. +- Enable testing of the full module deployment at every run. + +The default removal procedure works fine for most of the modules created so far, so it's likely you won't have to change anything to make the module you're editing to be removed correctly after deployment. + +# How it works + +The removal process will remove all resources created during deployment. The list is identified by: + +1. Recursively fetching the list of resource IDs created through your deployment (resources created by deployments created by the parent one will be fetched too) +1. Ordering the list based on resource IDs segment count (ensures child resources are removed first. E.g. `storageAccount/blobServices` comes before `storageAccount` as it has one more segments delimited by `/`) +1. Filtering out from the list any resource used as dependencies for different modules (e.g. the commonly used Log Analytics workspace) +1. Moving specific resource types to the top of the list (if a certain order is required). For example `vWAN` requires its `Virtual Hubs` to be removed first, even though they are no child-resources. + +After a resource is removed (this happens after each resource in the list), the script will execute, if defined, a **post removal operation**. This can be used for those resource types that requires a post processing, like purging a soft-deleted key vault. + +The procedure is initiated by the script `/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1`, run during deployment by: +- (Azure DevOps) `/.azuredevops/pipelineTemplates/module.jobs.deploy.yml` +- (GitHub) `/.github/actions/templates/validateModuleDeployment/action.yml` + +It uses several helper scripts that can be found in the `/utilities/pipelines/resourceRemoval/helper` folder +# Create a specialized removal procedure + +You can define a custom removal procedure by: +1. influencing the **order** in which resources are removed by prioritizing specific resource types + > **Example** Removing a _Virtual WAN_ resource requires related resources to be deleted in a specific order +1. defining a **custom removal action** to remove a resource of a _specific resource type_ + > **Example** A _Recovery Services Vault_ resource requires some protected items to be identified and removed beforehand +1. defining a custom **post-removal action** to be run after removing a resource of a _specific resource type_ + > **Example** A _Key Vault_ resource needs to be purged when soft deletion is enforced + +Those methods can be combined independently. + +> **Important**: _custom_ and _post-removal_ actions will be executed when a resource of the type you specify is removed **regardless** of which deployment triggered the deployment. Make sure you do not assume the resource is in a particular state defined by your module. + +To modify the resource types removal **order**: +1. Open the `/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1` file. +1. Look for the following comment: `### CODE LOCATION: Add custom removal sequence here` +1. Add a case value that matches your module name +1. In the case block, update the `$removalSequence` variable value to accommodate your module requirements +1. Remember to add the `break` statement. + +To defina a **custom removal** action: +1. Open the `/utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1` file. +1. Look for the following comment: `### CODE LOCATION: Add custom removal action here` +1. Add a case value that matches the resource type you want to modify the removal action for +1. In the case block, define the resource-type-specific removal action + +To add a **post-removal** step: +1. Open the `/utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1` file. +1. Look for the following comment: `### CODE LOCATION: Add custom post-removal operation here` +1. Add a case value that matches the resource type you want to add a post-removal operation for +1. In the case block, define the resource-type-specific post removal action diff --git a/docs/wiki/PipelinesDesign.md b/docs/wiki/PipelinesDesign.md index dbe0085441..bfe1ac39eb 100644 --- a/docs/wiki/PipelinesDesign.md +++ b/docs/wiki/PipelinesDesign.md @@ -93,9 +93,9 @@ Note that, for the deployments we have to account for certain [prerequisites](#p #### Removal -The removal phase is strongly coupled with the previous deployment phase. Fundamentally, we want to remove any test-deployed resource after its test concluded. If we would not, we would generate unnecessary costs and may temper with any subsequent test. Some resources may require a dedicated logic to be removed. This logic should be stored alongside the generally utilized removal script in the `.utilities/pipelines/resourceRemoval` folder and be referenced by the `Initialize-DeploymentRemoval.ps1` script that orchestrates the removal. +The removal phase takes care of removing all resources deployed as part of the previous deployment phase. The reason is twofold: keeping validation subscriptions costs down and allow deployments from scratch at every run. -Most of the removal scripts rely on the deployment name used during the preceding deployment step. Based on this name in combination with the template file path, the removal script find the corresponding deployment and removes all contained resources. +For additional details on how removal works please refer to the dedicated [Removal action](PipelineRemovalAction) page. ### Publish @@ -104,7 +104,7 @@ The publish phase concludes each module's pipeline. If all previous tests succee - _private bicep registry_ By the time of this writing, the publishing experience works as follows: -1. A user can optionally specific a specific version in the module's pipeline file, or during runtime. If the user does not, a default version is used +1. A user can optionally specify a version in the module's pipeline file, or during runtime. If the user does not, a default version is used 1. No matter what publishing location we enabled, the corresponding logic will 1. Fetch the latest version of this module in the target location (if available) 1. Compare it with any specified custom version the user optionally provided diff --git a/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1 b/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1 index bb4cc7166d..ec6c1f2af3 100644 --- a/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1 +++ b/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1 @@ -72,6 +72,7 @@ function Initialize-DeploymentRemoval { ) break } + ### CODE LOCATION: Add custom removal sequence here } # Invoke removal diff --git a/utilities/pipelines/resourceRemoval/helper/Get-ResourceIdsOfDeployment.ps1 b/utilities/pipelines/resourceRemoval/helper/Get-ResourceIdsOfDeployment.ps1 index cdb7dd31bd..4c7a389e9f 100644 --- a/utilities/pipelines/resourceRemoval/helper/Get-ResourceIdsOfDeployment.ps1 +++ b/utilities/pipelines/resourceRemoval/helper/Get-ResourceIdsOfDeployment.ps1 @@ -51,9 +51,9 @@ function Get-ResourceIdsOfDeploymentInner { 'resourcegroup' { if (Get-AzResourceGroup -Name $resourceGroupName -ErrorAction 'SilentlyContinue') { [array]$deploymentTargets = (Get-AzResourceGroupDeploymentOperation -DeploymentName $name -ResourceGroupName $resourceGroupName).TargetResource | Where-Object { $_ -ne $null } - foreach ($deployment in ($deploymentTargets | Where-Object { $_ -notmatch '/deployments/' } )) { - Write-Verbose ('Found deployment [{0}]' -f $deployment) -Verbose - [array]$resultSet += $deployment + foreach ($resourceId in ($deploymentTargets | Where-Object { $_ -notmatch '/deployments/' } )) { + Write-Verbose ('Found resource [{0}]' -f $resourceId) -Verbose + [array]$resultSet += $resourceId } foreach ($deployment in ($deploymentTargets | Where-Object { $_ -match '/deployments/' } )) { $name = Split-Path $deployment -Leaf diff --git a/utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1 b/utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1 index a1757e152d..907d99c855 100644 --- a/utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1 +++ b/utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1 @@ -109,5 +109,6 @@ function Invoke-ResourcePostRemoval { # Undo a potential soft delete state change $null = Set-AzRecoveryServicesVaultProperty -VaultId $vaultId -SoftDeleteFeatureState $softDeleteStatus.TrimEnd('d') } + ### CODE LOCATION: Add custom post-removal operation here } } diff --git a/utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1 b/utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1 index 1fd2854397..88510732e2 100644 --- a/utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1 +++ b/utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1 @@ -95,6 +95,7 @@ function Invoke-ResourceRemoval { # -------------- $null = Remove-AzResource -ResourceId $resourceId -Force -ErrorAction 'Stop' } + ### CODE LOCATION: Add custom removal action here Default { $null = Remove-AzResource -ResourceId $resourceId -Force -ErrorAction 'Stop' } diff --git a/utilities/pipelines/resourceRemoval/helper/Remove-Deployment.ps1 b/utilities/pipelines/resourceRemoval/helper/Remove-Deployment.ps1 index 3137192c12..171a525b35 100644 --- a/utilities/pipelines/resourceRemoval/helper/Remove-Deployment.ps1 +++ b/utilities/pipelines/resourceRemoval/helper/Remove-Deployment.ps1 @@ -22,7 +22,7 @@ Optional. The time to wait in between the search for resources via their remove Optional. The deployment name to use for the removal .PARAMETER TemplateFilePath -Optional. The path to the deployment file +Mandatory. The path to the deployment file .PARAMETER RemovalSequence Optional. The order of resource types to apply for deletion