Azure Monitor Baseline Alerts for Platform Observability - Part 2

Stephen Tulp
December 18, 2024

15 minutes to read

Tags: azureiacPlatformEngineeringGitHub

Building on yesterday, the Azure Monitor Baseline Alerts (AMBA) for Azure Landing Zones (ALZ) solution is a best practice collection of alerts for resources commonly deployed in Azure landing zones. It demonstrates how to deploy alerts at scale using Azure Policy.

AMBA includes code and detailed instructions for the following:

• Azure Policy Definitions to automatically create alerts, action groups, and alert processing rules for Azure resource types, based on a recommended Azure Monitor Baseline for Alerting in either a green fields or brown fields environment.
• Azure Policy Initiatives that group these policies into appropriate categories for easier policy assignment, aligned with the ALZ Platform structure.

Creation of alerts, action groups, and alert processing rules follow this structure:

1. Metric Alerts get created in the resource group where the monitored resource resides. For instance, if creation of an ER circuit is in a resource group governed by the policies, the corresponding alerts will be in that same resource group.
2. Activity Log Alerts get created in a designated resource group (specifically created and used for this solution) within each Landing Zone Subscription upon deployment. The resource group name has a default value of rg-amba-monitoring-001, which is a configurable parameter.
3. Resource Health Alerts get created in a designated resource group (specifically created and used for this solution) within each Landing Zone Subscription upon deployment. The resource group name has a default value of rg-amba-monitoring-001, which is a configurable parameter.
4. Action Groups and Alert Processing Rules get created in a designated resource group (specifically created and used for this solution) within each Landing Zone Subscription upon deployment. The resource group name has a default value of rg-amba-monitoring-001, which is a configurable parameter.

Prerequisites

The following pre-requisites need to be in place first.

1. A Microsoft Entra Id Tenant, this should be easy if you are using Azure already.
2. An Azure Landing Zone aligned Management group hierarchy deployed as outlined in the Azure landing zone design areas and conceptual architecture documentation. We will be using the structure from Part 1 of this series.
3. At least one Azure Subscription for deploying alerts through policies.
4. A Deployment Identity with Owner permissions to the pseudo root management group. This permission is necessary for the Service Principal Account to create role-based access control assignments.
5. If deploying manually via Azure CLI or PowerShell, ensure you have Bicep installed. Refer to the configuration guides for Azure CLI and PowerShell.
6. The following Azure resource providers registrations on all subscriptions in scope for the policies to function:
   • Microsoft.AlertsManagement
   • Microsoft.Insights
7. To utilise log alerts for virtual machines (both Azure and Azure Arc), ensure that you have VM Insights enabled.

After completing the pre-requisites then we can…

• Fork this repository to the internal GitHub or Azure DevOps organisation.
• Clone the repository from there to your local development environment.
• There are five (5) ways to deploy (Portal Accelerator, GitHub Actions, Azure DevOps Pipelines, Azure CLI or PowerShell), we will deploy the following to showcase the deployment.
  • Deploy via the Azure Portal (Preview)
  • Automated deployment with GitHub Actions

Azure Portal Accelerator

In preview, the AMBA Azure Portal Accelerator provides a visual representation of the deployment and surfaces configurable parameters that enables the customisation of the solution.

Deployment Settings Blade

• Select the Pseudo Root Management Group that you want to use: Insight - Azure Landing Zones
• Choose the value of Region to specify the Azure location: Australia East
• Change the value of Resource group for baseline alerts to the name of the resource group where the activity logs, resource health alerts, actions groups and alert processing rules will live: AlertsRG
• Choose the value of Resource group location to specify the resource group location: Australia East
• Choose the value of Bring Your Own User Assigned Managed Identity to specify if you want to bring your own user assigned managed identity for monitoring purpose: No
• Define the value of User Assigned Managed Identity Name to specify the name of the user assigned managed identity for monitoring purpose: umi-syd-amba-prd-001
• Choose the value of Management Subscription Id to specify the subscription Id where the user assigned managed identity will be created: Stephen Tulp Azure MVP
• Choose the value of Customer Usage Selection Option Microsoft can identify the deployments of the Azure Resource Manager and Bicep templates with the deployed Azure resources: Enabled
• Change the value of Resource Group Tags to specify the tags on the resource group: _deployed_by_amba: true

Management Groups Settings Blade

• Choose the value of Enterprise Scale Company Management Group to the management group Id for Platform: mg-alz1
• Choose the value of Platform Management Group to the management group Id for Platform: mg-alz1-platform
• Choose the value of Connectivity Management Group to the management group Id for Connectivity: mg-alz1-platform-connectivity
• Choose the value of Identity Management Group to the management group Id for Identity: mg-alz1-platform-identity
• Choose the value of Management Management Group to the management group Id for Management: mg-alz1-platform-management
• Choose the value of Landing Zone Management Group to the management group Id for Landing Zones: mg-alz1-landingzones
• Choose the value of Enable AMBA Connectivity This initiative deploys Azure Monitor Baseline Alerts for Platform network services: Yes
• Choose the value of Enable AMBA Identity This initiative deploys Azure Monitor Baseline Alerts for Identity services: Yes
• Choose the value of Enable AMBA Management This initiative deploys Azure Monitor Baseline Alerts for Management services: Yes
• Choose the value of Enable AMBA Hybrid VM This initiative deploys Azure Monitor Baseline Alerts for Azure Arc-enabled Servers: Yes
• Choose the value of Enable AMBA Azure VM This initiative deploys Azure Monitor Baseline Alerts for Azure Virtual Machines: Yes
• Choose the value of Enable AMBA Key Management This initiative deploys Azure Monitor Baseline Alerts for Key Management Services: Yes
• Choose the value of Enable AMBA Load Balancing This initiative deploys Azure Monitor Baseline Alerts for Load Balancing Services: Yes
• Choose the value of Enable AMBA Network Changes This initiative implements Azure Monitor Baseline Alerts for alterations in Network Routing and Security: Yes
• Choose the value of Enable AMBA Recovery Services This initiative deploys Azure Monitor Baseline Alerts for Recovery Services: Yes
• Choose the value of Enable AMBA Storage This initiative deploys Azure Monitor Baseline Alerts for Storage Services such as Storage accounts: Yes
• Choose the value of Enable AMBA Web This initiative deploys Azure Monitor Baseline Alerts for Web Services such as App Services: Yes
• Choose the value of Enable AMBA Service Health This initiative deploys Azure Monitor Baseline Alerts for Service Health Events: Yes
• Choose the value of Enable AMBA Notification Assets This configuration will deploy notification assets broad notifications: Yes

Notification Settings Blade

• Choose the value of Bring Your Own Notifications (BYON) if you want to use existing Action Groups and Alert Processing Rules: No
• Specify the email address(es) for Email contact for action group notifications to receive notifications for alerts, including Service Health alerts: [“test@test.com”]
• Leave Webhook Service Uri, ARM Role Id and Add Logic App/Eventhub/Azure Function as action: No

Review and Finalise

Once all the settings and fields are complete, navigate to the Review + Create tab and create the deployment.

GitHub Actions Deployment

The deployment for GitHub Actions is straightforward and uses the Az CLI to do a Management Group deployment at the Pseudo Root Management Group to deploy the Policy Definitions, Initiatives and Assignments.

GitHub Action Workflow

We will use a GitHub Actions Workflow example, that will require some minor tweaking, refer to the Deploy AMBA.yml.

name: Deploy AMBA

on:
  workflow_dispatch: {}

permissions:
  id-token: write
  contents: read

env:
  Location: "australiaeast"
  ManagementGroupPrefix: "mg-alz"

jobs:
  deploy_job:
    runs-on: ubuntu-latest
    environment: deploy

    steps:
      # Checkout the repository
      - name: Checkout Repo
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      # Login to Azure
      - name: "Az CLI login"
        uses: azure/login@v1
        with:
          client-id: $
          tenant-id: $
          subscription-id: $
          enable-AzPSSession: true

      # Deploy AMBA ARM template
      - name: Az CLI Deploy
        id: deploy_amba
        shell: bash
        run: |
          az deployment mg create --template-uri https://raw.githubusercontent.com/Azure/azure-monitor-baseline-alerts/main/patterns/alz/alzArm.json --location $ --management-group-id $ --parameters ./patterns/alz/alzArm.param.json

Configure and update the following values in the workflow based on what you want to achieve.

• Configure OpenID Connect to allow the Az CLI login.
• Change the Environment variables
  • Change Location: to the preferred Azure region.
  • Change ManagementGroupPrefix: to the pseudo root management group Id that represents the hierarchy in a CAF-aligned Management Group structure.
• We are using the templateUri property, this can also be local if required.
• Update the parameters path to where the .json file is.
• Save the file to the .github/workflow folder.

Parameter File

A sample parameter file is available in the repo, refer to AMBA Parameter File, the parameter file is large (almost 3000 lines) as it has configurable values for all the policies.

The first 120 lines is where most parameters require changing and when you are testing and understanding the solution this is where i would start. Most of these values are the configurable inputs that we addressed in the portal deployment.

    "enterpriseScaleCompanyPrefix": {
      "value": "mg-alz1"
    },
    "platformManagementGroup": {
      "value": "mg-alz1-platform"
    },
    "IdentityManagementGroup": {
      "value": "mg-alz1-platform-identity"
    },
    "managementManagementGroup": {
      "value": "mg-alz1-platform-management"
    },
    "connectivityManagementGroup": {
      "value": "mg-alz1-platform-connectivity"
    },
    "LandingZoneManagementGroup": {
      "value": "mg-alz1-landingzones"
    },
    "enableAMBAConnectivity": {
      "value": "Yes"
    },
    "enableAMBAIdentity": {
      "value": "Yes"
    },
    "enableAMBAManagement": {
      "value": "Yes"
    },
    "enableAMBAServiceHealth": {
      "value": "Yes"
    },
    "enableAMBANotificationAssets": {
      "value": "Yes"
    },
    "enableAMBAHybridVM": {
      "value": "Yes"
    },
    "enableAMBAKeyManagement": {
      "value": "Yes"
    },
    "enableAMBALoadBalancing": {
      "value": "Yes"
    },
    "enableAMBANetworkChanges": {
      "value": "Yes"
    },
    "enableAMBARecoveryServices": {
      "value": "Yes"
    },
    "enableAMBAStorage": {
      "value": "Yes"
    },
    "enableAMBAVM": {
      "value": "Yes"
    },
    "enableAMBAWeb": {
      "value": "Yes"
    },
    "telemetryOptOut": {
      "value": "No"
    },
    "bringYourOwnUserAssignedManagedIdentity": {
      "value": "No"
    },
    "bringYourOwnUserAssignedManagedIdentityResourceId": {
      "value": ""
    },
    "userAssignedManagedIdentityName": {
      "value": "umi-syd-amba-prd-001"
    },
    "managementSubscriptionId": {
      "value": "0b5d0018-2879-4810-b8d7-4f8dda5ce0b9"
    },
    "ALZMonitorResourceGroupName": {
      "value": "rg-amba-monitoring-001"
    },
    "ALZMonitorResourceGroupLocation": {
      "value": "australiaeast"
    },
    "ALZMonitorResourceGroupTags": {
      "value": {
        "Project": "amba-monitoring"
      }
    },
    "ALZMonitorDisableTagName": {
      "value": "MonitorDisable"
    },
    "ALZMonitorDisableTagValues": {
      "value": [
        "true",
        "Test",
        "Dev",
        "Sandbox"
      ]
    },
    "ALZMonitorActionGroupEmail": {
      "value": ["test@test.com"]
    },
    "ALZLogicappResourceId": {
      "value": ""
    },
    "ALZLogicappCallbackUrl": {
      "value": ""
    },
    "ALZArmRoleId": {
      "value": []
    },
    "ALZEventHubResourceId": {
      "value": []
    },
    "ALZWebhookServiceUri": {
      "value": []
    },
    "ALZFunctionResourceId": {
      "value": ""
    },
    "ALZFunctionTriggerUrl": {
      "value": ""
    },
    "BYOActionGroup": {
      "value": []
    },
    "BYOAlertProcessingRule": {
      "value": ""
    },

Reviewing the Deployment Output

Regardless of the deployment approach taken, you can see the individual deployments and modules that are deployed. Navigating to the Deployment pane on the Pseudo Root Management Group will show the following:

You can also see the Policy Definitions, Initiatives and Assignments that are created as part of the deployment and which Management Group they are assigned to.

Looking at the Deploy Azure Monitor Baseline Alerts for Key Management Policy Assignment for example, you can see the defined parameter Ids and names and also the corresponding parameter value.

If we look in the AlertsRg Resource Group we will be able to see the generated Activity Log Alert Rules, Action Groups, and Alert Processing Rules

Then if we look where the Azure resources are hosted, we will see the associated Metric Alert Rules for each support resource type. Below we can see Metric Alert Rules rules for Platform Connectivity resources, such as Azure Firewall Policy, Public IP Addresses and the virtual network

Conclusion

Azure Monitor Baseline Alerts (AMBA) is an essential tool for enhancing the monitoring capabilities of Azure services. It offers predefined metrics and alert rules that help organisations maintain optimal performance and reliability of their Azure resources. By using AMBA, users can receive near real-time notifications about potential issues, visualise alerts through intuitive dashboards, and automate alert policies with Azure Policy templates.

This proactive monitoring approach ensures early problem detection, reduces downtime, and optimises resource utilisation, leading to a more efficient and resilient Azure estate.

Share on Twitter (X) Share on LinkedIn RSS Feed