Insight Tech APAC Blog Logo

Anti-Pattern 5: Passing secrets from pipelines without first storing them in a Key Vault

trentsteenholdt
December 5, 2024

11 minutes to read

Tags: azureiacbicepanti-patternssecuritysecrets

Azure Bicep Anti-Patterns

Welcome to the fifth installment of the series on Azure Bicep anti-patterns. Today, we address a critical security concern: passing secrets from pipelines without first storing them in a Key Vault. Proper secret management is paramount to maintaining secure and reliable infrastructure deployments especially when triggered from CI/CD pipeline tools like GitHub Actions, Azure Pipelines, Octopus Deploy and so on. Ignoring this practice can lead to vulnerabilities, complicate your deployment processes and actually make your overall mean time to recovery (MTTR) a much higher number then it needs to be.

The Anti-Pattern: Passing Secrets Directly Through Pipelines

A prevalent mistake in Azure Bicep deployments is passing secrets directly through CI/CD pipelines without utilising Azure Key Vault. This approach can manifest in several ways:

  • Hard-Coding Secrets: Embedding secrets directly in Bicep templates or pipeline scripts.
  • Environment Variables: Exposing secrets as plain environment variables that can be accessed by unauthorised entities.
  • Insecure Storage: Using unsecured storage mechanisms for secrets, making them vulnerable to breaches.

Why is this a problem

  • Security Risks: Exposed secrets can be exploited by malicious actors, leading to unauthorised access and data breaches.
  • Compliance Issues: Many industries have strict regulations regarding the handling of sensitive information. Ignoring best practices can lead to non-compliance.
  • Maintenance Challenges: Managing secrets directly within pipelines or templates increases the risk of accidental leaks, forgetting what the secret actually was and complicates secret rotation processes.
  • Auditability: There is no easy way in CI/CD tools to track easily when secrets/variables changes and by whom. This can result in passwords being replaced from the source with no understanding when this was done and why. This can also lead to issues where no one knows the secret, and therefore no one knows the password of that’s set in Azure! You’ll be surprised just how often we see this.

Example of Insecure Secret Handling

Consider the following insecure approach where a secret is passed directly through a pipeline to a Bicep template:

# Azure Pipelines example
- stage: DeployInfrastructure
  jobs:
    - job: DeployBicep
      steps:
        - script: |
            az deployment group create --template-file main.bicep --parameters adminPassword=$(AdminPassword)
          env:
            AdminPassword: $(ADMIN_PASSWORD)

In this example:

  • The AdminPassword is stored as a pipeline variable and passed directly to the Bicep template.
  • If the pipeline logs are not properly secured, the secret could be exposed in plain text!

The Solution: Leveraging Azure Key Vault and Secret Variables in your CI/CD Tooling

Azure Key Vault provides secure storage of secrets, keys, and certificates. Integrating Key Vault into your deployment pipelines ensures that secrets are managed securely and efficiently.

Benefits of Using Azure Key Vault

  • Enhanced Security: Secrets are stored securely with access controls and auditing. This also allows for humans to securely obtain a secret/password through secure means from the Azure Key Vault itself reducing access to people setting the secrets in the CI/CD tooling.
  • Centralised Management: Manage all secrets in a single location, simplifying secret rotation and lifecycle management. E.g. You can apply Azure Policy on the Azure Key Vault to ensure secrets are rotated etc. for enhanced security.
  • Integration with Azure Services: Seamlessly integrate with Azure Bicep, PowerShell, and Azure CLI for streamlined deployments. Make use of functions like getSecret() in Bicep.

Implementing Key Vault in Your Bicep Deployments

Step 1: Create a Key Vault with Bicep

Create your first part of your pipeline to create an Azure KeyVault.

# Azure Pipelines example
- variables:
    group: my-group # This is where the AdminPassword is stored in a secure variable.

- stage: DeployInfrastructure
  jobs:
    - job: DeployBicep
      steps:
        - script: |
            az deployment group create --template-file keyvault.bicep 


// keyVault.bicep
resource keyVault 'Microsoft.KeyVault/vaults@2021-06-01-preview' = {
  name: 'mySecureKeyVault'
  location: location
  properties: {
    sku: {
      family: 'A'
      name: 'standard'
    }
    tenantId: subscription().tenantId
    accessPolicies: []
  }
}

output keyVaultId string = keyVault.id
output keyVaultName string = keyVault.name

Step 2: Push Secrets to Key Vault in the Pipeline

Use a separate Bicep deployment to push secrets to Key Vault before other deployments depend on them. This ensures that secrets are managed centrally and securely.

# Azure Pipelines example
- stage: PushSecrets
  jobs:
    - job: PushToKeyVault
      steps:
        - task: AzureCLI@2
          inputs:
            azureSubscription: 'AzureServiceConnection'
            scriptType: 'ps'
            scriptLocation: 'inlineScript'
            inlineScript: |
              # Push the secret to Key Vault using Bicep deployment
              az deployment group create --template-file deploySecrets.bicep --parameters keyVaultId=$(KEYVAULT_ID) adminPassword=$(AdminPassword)

// deploySecrets.bicep
param keyVaultId string
param adminPassword string

resource keyVault 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  id: keyVaultId
}

resource adminSecret 'Microsoft.KeyVault/vaults/secrets@2021-06-01-preview' = {
  name: 'AdminPassword'
  parent: keyVault
  properties: {
    value: adminPassword
  }
}

output adminSecretName string = adminSecret.name

Step 3: Reference Key Vault Secrets in Bicep

Add to your Bicep templates to retrieve secrets from Key Vault and use them in your resources, such as a Virtual Machine.

// vmDeployment.bicep
param keyVaultId string

resource keyVault 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  id: keyVaultId
}

resource vm 'Microsoft.Compute/virtualMachines@2021-07-01' = {
  name: 'myVM'
  location: location
  properties: {
    hardwareProfile: {
      vmSize: 'Standard_DS1_v2'
    }
    osProfile: {
      computerName: 'myVM'
      adminUsername: 'azureUser'
      adminPassword: keyVault.getSecret('AdminPassword')
    }
    networkProfile: {
      networkInterfaces: [
        {
          id: nic.id
        }
      ]
    }
  }
}

resource nic 'Microsoft.Network/networkInterfaces@2021-05-01' = {
  name: 'myNIC'
  location: location
  properties: {
    ipConfigurations: [
      {
        name: 'ipconfig1'
        properties: {
          subnet: {
            id: subnet.id
          }
          privateIPAllocationMethod: 'Dynamic'
        }
      }
    ]
  }
}

resource subnet 'Microsoft.Network/virtualNetworks/subnets@2021-05-01' existing = {
  name: 'myVNet/mySubnet'
}

See Retrieve secrets in Bicep file on the use of getSecret() in Bicep

Integrating Key Vault with CI/CD Pipelines

The Chicken and Egg Problem

In Azure DevOps Pipelines and other CI/CD tools like Octopus Deploy, there is usually the option to store secrets/variable directly into an Azure Key Vault at the first instance. That is, store directly in Azure Key Vault and don’t store in the CI/CD tool’s back end. Personally, I frown upon this approach as integrating Azure Key Vault directly into CI/CD Tools can create a chicken and egg problem: the Azure Key Vault needs to exist before the pipeline can run. This circular dependency complicates the initial deployment and can hinder end-to-end automation.

Instead of embedding Key Vault integration within the pipeline, I always recommend to first push secrets to Key Vault using pipeline secrets, and then reference these secrets in your Bicep templates using the getSecret function.

This approach ensures that Key Vault is populated with secrets before the Bicep deployment relies on them, avoiding the chicken and egg problem.

Best Practices for Secret Management

  1. Always Use Key Vault for Secrets: Never hard-code or pass secrets directly through pipelines to a resource like a Virtual Machine, SQL Servers, Application Gateway etc.
  2. Restrict Access: Implement strict role based access controls in Key Vault to ensure only authorised entities can retrieve secrets.
  3. Enable Auditing: Monitor and audit access to Key Vault to detect any unauthorised attempts.
  4. Automate Secret Rotation: Regularly rotate secrets to minimise the risk of compromised credentials. Or worse, credentials being taken from the Key Vault and store in a place like it shouldn’t be, e.g. a teams message!
  5. Use Managed Identities: Leverage Azure Managed Identities to allow services to authenticate to Key Vault without managing credentials.

Example: Using Managed Identity to Access Key Vault

// main.bicep

module keyVaultModule './keyVault.bicep' = {
  name: 'deployKeyVault'
  params: {
    location: location
  }
}

resource appService 'Microsoft.Web/sites@2021-02-01' = {
  name: 'myAppService'
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    // App Service properties
  }
}

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(keyVaultModule.outputs.keyVaultId, appService.id, 'b86a8e5e-7d3a-4b6e-9b9b-1d7b6a8a5b6e') // Unique GUID for the role assignment
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b86a8e5e-7d3a-4b6e-9b9b-1d7b6a8a5b6e') // Key Vault Secrets User role
    principalId: appService.identity.principalId
    principalType: 'ServicePrincipal'
    scope: keyVaultModule.outputs.keyVaultId
  }
}

output appServicePrincipalId string = appService.identity.principalId

In this example:

  • App Service: Configured with a system-assigned managed identity.
  • Role Based Access Control: Grants the App Service permissions to access secrets in Key Vault using the Key Vault Secrets User role.

Conclusion

Effective secret management is a cornerstone of secure and maintainable Infrastructure as Code practices. By not passing secrets directly through pipelines and instead leveraging Azure Key Vault, you ensure that sensitive information remains protected, reducing the risk of security breaches and enhancing the overall reliability of your deployments. You also make it possible for the secret to be pulled by humans in the times of disaster recovery easily.

What’s Next?

In our next post of this series, we’ll delve into the importance of properly implementing RBAC and security practices in your IaC projects. Neglecting these areas can undermine your infrastructure’s integrity and expose it to potential threats. Stay tuned as we continue to explore common anti-patterns and how to avoid them.