Microsoft Symphony for Engineering & Platform Orchestration
Stephen Tulp
December 11, 2024
8 minutes to read
Introduction
Symphony is a framework, set of patterns, and best practices for developing, testing, and deploying infrastructure on Azure using Infrastructure as Code (IaC). It includes modern DevOps practices for IaC, such as Main and Pull Request workflows, IaC Code Validation and Lining, Automated Testing, Security Scanning, Multi-environment deployments, and module dependencies.
Symphony is an enterprise-level CI/CD multi-orchestrator, a multi-IaC framework that encapsulates best practices and guidelines for developing, testing, and deploying infrastructure as code, allowing preparation and deployment to production quickly.
Symphony supports both Terraform and Bicep for IaC and either Azure DevOps or GitHub as the Build Orchestrator.
Symphony Value Proposition
Symphony offers workflows to implement IaC automation and provides multi-environment support that can be used on both public and non-public clouds. With a modular workflow design, it integrates with other tools with no need to update the workflow or change the CI/CD pipeline.
Some of the key features include:
| Feature | Description |
| Security Scanning | Symphony helps prevent credential leaking in the IaC code by running security scanners in the workflows. |
| Linting & Validation | Symphony ensures no invalid IaC code early and reduces the development iteration loop. |
| IaC Modules Dependency | Symphony offers a clear structure to organise modules based on dependency while allowing for the flexibility of automatically passing outputs from one module to the next. |
| Modules & End-to-end testing | Symphony provides samples to write, execute, and report on module tests and end-to-end tests for the IaC modules. |
| Multi Environment support | Symphony offers a clear pattern to store different IaC module configurations per environment, allowing the workflows to swap configs based on the target environments. |
Symphony comes with a devContainer to provide developers with a consistent, isolated, and reproducible development environment. This eliminates the need for manual configuration and management of the environment, making it easy to set up and maintain and includes all the required software as outlined here
Orchestration Workflows
Creating pipelines for IaC might seem straightforward, but can become complex due to the need to manage many dynamic components. A robust IaC workflow automates resource deployment but also integrates engineering principles, resource validation, dependency management, test execution, security scanning, and more.
Symphony provides multiple workflows to ensure engineering excellence throughout the IaC process. These workflows are orchestrator-independent, relying on logic implemented in bash scripts, with pipelines serving as thin layers that execute these scripts. This design simplifies updating workflows or integrating new tools, regardless of the source control management system, by adding them to the workflow scripts.
Lets break down the Bicep andGitHub workflows to get a better understanding of the stage and tasks that are executed.
Deploy Workflow
This Deploy CI workflow is designed to deploy IaC resources to the specified environment and subscription, adhering to best practices in IaC code repositories. It features multi-stage tasks to validate, deploy, and test all necessary resources, and to report any failures. The specifics of each stage’s execution may vary depending on the capabilities of the orchestrator’s IaC tool.
The workflow includes a set of input variables that are updated at execution time. While these inputs may differ based on the chosen IaC tool, a common input variable across all is the environment name, which defines the environment configurations used.”
Validate
This stage ensures code readiness and executes pre-validation events, runs validations and linting tools, scans code for possible cred leaks, and executes any unit tests. Stage steps execute in the following sequential order.
Preview and Deploy
This stage plans the execution of the IaC code and estimates the scope of the changes. It initializes the IaC tool selected, runs plan/what-if commands to detect the changing scope, executes pre_deploy events, then runs deploy commands to update the resources, executes post_deploy events, and eventually ensures successful resource updates.
Test
This stage executes the integration or end-to-end tests against the recently deployed/updated resources to ensure the configurations/changes are reflected and resources are working as expected. It then publishes the results of the tests and drops them as artifacts for future reference.
Tests are completed using arm-ttk and Pester for Bicep and Terratest for Terraform.
Pull Request Workflow
The PR Workflow offers a set of validations to vet the pull request branch code changes as part of the review process. This will ensure changes of the IaC are validated at the code quality level but also been properly tested on a real deployed environment to see the impact of resource changes.
The steps below outlines tasks undertaken as part of each stage.
Workflow tools
Symphony workflows use a set of tools to run tasks at each stage. Those tools are automatically installed and configured by the workflow tasks as needed during the execution of the workflow tasks.
- Azure CLI used to authenticate to the targeted environment Azure subscription.
- Gitleaks used to scan and prevent secrets from getting leaked into the main branch. Some alternatives to Gitleaks are, Aqua Vulnerability Scanning and Management, GitHub Code Scanning, Mend SAST, Oxeye, Sonar Cloud
Bicep workflow tools
- Bicep used to deploy the Bicep IaC modules.
- Powershell Core used to execute PowerShell scripts, like ARM-TTK scripts.
- ARM-TTK used to validate ARM templates.
- Shellspec used to write and execute tests to test Bicep modules.
- Pester used to write and execute tests to test Bicep modules.
Terraform workflow tools
- Terraform used to deploy IaC modules resources.
- TFLint used to validate the Terraform modules code.
- Go used to write and execute tests for the terraform modules code using Terratest.
Thoughts
While the Symphony framework provide a good approach to CI/CD for IaC, a fellow colleague pointed out that doing the Test stage after the deploy stage is not recommended and actually an anti-pattern. He talks about this and other anti-patterns in this blog post Not doing software development lifecycle (SDLC) practices with IaC
The diagram below outlines the Insight approach to CI/CD using GitHub
- A
Branchis created fromMain - Changes get committed to the
Branch - Once the changes are complete, a
PRis created - This triggers the
PR.yml - The
PRwill require approval from a codeowner - Once approved this will trigger the
Release & Deployworkflows.
These GitHub workflows are in the Azure Platform Engineering Advent Calendar 2024
Conclusion
In conclusion, the Symphony framework stands out as a powerful tool for Platform Engineering teams to provide to developers, offering a streamlined and efficient development experience. By leveraging DevContainers, Symphony ensures a consistent, isolated, and reproducible environment, which reduces setup time and enhances code quality. This approach simplifies the development process but also fosters a more productive and collaborative workflow. With Symphony, developers can focus more on innovation and less on configuration, ultimately driving better outcomes for their projects.