Setting up Terraform in Azure DevOps

Introduction

Terraform is an Infrastructure as a Code tool where you can deploy Cloud resources from several Cloud Service Providers such as a Microsoft Azure, Amazon AWS, Google Cloud. There are several guides on how Terraform works, but most of them are done by performing Terraform commands on your local devices (CMD or Powershell).

In the real world people use Azure DevOps as an example to deploy their Terraform code to their infrastructure.

This blog describes how you can enroll Terraform in your Azure environment via Azure DevOps.

Prerequisites

Service Connection to your Azure Subscription via Azure DevOps – To deploy Azure Resources on Azure
(Self) Hosted Agent on Azure DevOps – To deploy the pipeline code
Terraform basic knowledge (init, plan, and apply) – To understand how Terraform works
IDE (e.g. Visual Studio Code) – to write your code to Azure DevOps Repos

Setting up Terraform in Azure DevOps

Go to the Azure DevOps Marketplace and download Terraform by Microsoft DevLabs

Setting up the Terraform files

2. Create these files & folders shown below (or you can download all the files here)

3. Add the tfstate parameters in your dev.tfvars file as this is needed to refer your tfstate file in Terraform.

As for my example, I have used

Resource Group Name: rg-onemanitarmy-tfstate
Storage Account Name: stonemanitarmy
Blob Container Name: tfstate
Tfstate file key name: stonemanitarmy.terraform.tfstate
Remember these variables as you will need to mention these in step 10!

backend = {
  backendResourceGroupName                 = "<your tfstate resource group name>"
  backendStorageAccountName                = "<your tfstate storage account name>"
  backendContainerName                     = "<your tfstate blob container name>"
  backendKey                               = "<your tfstate backendkey name>.terraform.tfstate"
}

4. Create a Resource Group at main.tf as HelloWorld exercise

resource "azurerm_resource_group" "rg-helloworld" {
name = "rg-helloworld"
location = "West Europe"
}

5. Add the mandatory code in providers.tf to run Terraform

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.80.0"
    }
  }
    backend "azurerm" {
      resource_group_name  = var.backendResourceGroupName
      storage_account_name = var.backendStorageAccountName
      container_name       = var.backendContainerName
      key                  = var.backendKey
  }
}

# Configure the Microsoft Azure Provider
provider "azurerm" {
  skip_provider_registration = true 
  features {}
}

6. Add tfstate variables in variables.tf that is needed for running the pipeline.

variable "backendResourceGroupName" {
  description = "Terraform Backend Resource Group"
  type        = string
}

variable "backendStorageAccountName" {
  description = "Terraform Backend Storage Account"
  type        = string
}

variable "backendContainerName" {
  description = "Terraform Backend Container Name"
  type        = string
}

variable "backendKey" {
  description = "Terraform Backend Key"
  type        = string
}

variable "location" {
  default = "West Europe"
}

Setting up Azure Pipeline files

10. Paste this code below in the terraform-preparation.yaml file and replace the variable values <> with your own (including the ones in step 3 regarding resource group name and storage account name).

Context about the code
This code (via Powershell) creates a Storage Account and a Blob Storage Container to store the .tfstate file there.
Reason to store .tfstate file there is to store the .tfstate file centrally and to mitigate Git Conflicts in Azure Repos.

# Preparations file to create Storage Account in advance to save .tfstate file.
# Code in a nutshell:
  # Select Service Connection
  # Register Resource Provider Microsoft.Storage to create Storage resources in Azure.
  # Create Resource Group
  # Create Storage Account
  # Create Blob Container and store Terraform .tfstate file.

variables:
    ServiceConnectionName: "<fill in your service connection name>"                 # Specify service connection name in Azure DevOps project settings.
    SubscriptionName: "<fill in your subscription name>"                            # Specify Subscription name.
    ResourceGroupName: "<fill in your resource group name store to .tfstate file>"  # Specify Resource Group name.
    StorageAccountName: "<fill in your new storage account name>"                   # Specify Storage Account name.
    Location: "<fill in your location>"                                             # Specify Location.

trigger:
  - none

pool:
  vmImage: "ubuntu-latest"


stages:
  - stage: TerraformState
    jobs:
    - job: Create_Storage_Account
      continueOnError: false
      steps:
        - task: AzurePowerShell@5
          displayName: 'Create Storage Account for Terraform State'
          inputs:
            azureSubscription: '$(ServiceConnectionName)' # Use Service Connection to deploy the code to Azure.
            ScriptType: 'InlineScript'
            Inline: |                                     # Powershell Inline commands
              # Select Subscription to use
              Select-AzSubscription -SubscriptionName '$(SubscriptionName)'

              # Try to enable Resource Providers. 
              Try {
              Register-AzResourceProvider -ProviderNamespace Microsoft.Storage
              }

              # If already enabled, then send message that it is already enabled.
              Catch {
              echo "Resource Provider is already enabled."
              }

              # Create Resource Group.
              New-AzResourceGroup -Name '$(ResourceGroupName)' -Location '$(Location)' -Force

              # Create Storage Account
              New-AzStorageAccount -ResourceGroupName '$(ResourceGroupName)' -Name '$(StorageAccountName)' -Location '$(Location)' -SkuName Standard_LRS -Kind StorageV2
              
              # Select Storage Account and create Blob Container.
              $context = New-AzStorageContext -StorageAccountName '$(StorageAccountName)' -UseConnectedAccount
              New-AzStorageContainer -Name 'tfstate' -Context $context
              
              # Select Storage Account and disabled Public Network Access.
              Set-AzStorageAccount -ResourceGroupName '$(ResourceGroupName)' -Name '$(StorageAccountName)' -PublicNetworkAccess Disabled
              
              # Send message that the pipeline ran succesfully.
              echo "Succesfully created Resource Provider, Resource Group, Storage Account, and Blob Container with .tfstate."

            azurePowerShellVersion: 'LatestVersion'

11. Run the pipeline file and you will now have a resource group, storage account, and blob container in your Azure environment.

12. Create a Pipeline Environment in Pipelines > Environments before running plan script with the name ‘gen-terraform-env’.
This is needed to run the Terraform Plan and Apply script.

13. Add .gitignore file in the repository:

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log

# Exclude all .tfvars files, which are likely to contain sentitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
#
#*.tfvars

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

Terraform Plan – terraform-plan.yaml

14. Paste the pipeline code in your terraform-plan.yaml file and change the variables that you mentioned created in the earlier steps.

# Terraform PLAN Pipeline configuration

parameters:
- name: environment
  displayName: Environment OneManITArmy
  type: string
  values:
  - dev

# Create if statement to decide which environment Terraform needs to deploy.
variables:
  ${{ if eq(parameters.environment, 'dev') }}:
    serviceConnectionName: "<your service connection name>"                                      # Specify service connection name in Azure DevOps project settings.
    workingDirectory: '$(System.DefaultWorkingDirectory)'                                        # Specify working directory of your Terraform files.
    varFile: "$(System.DefaultWorkingDirectory)/env-tfvars/${{ parameters.environment }}.tfvars"
    backendResourceGroupName: "<your resource group name>"                                       # Specify RG-name that is created via Terraform where Storage account is held.
    backendStorageAccountName: "<your new storage account name>"                                 # Specify name of the Storage account is that created.
    backendContainerName: "<your tfstate blob container name>"                                   # Specify name of blob container in Storage account.
    backendKey: "<your tfstate backendkey name>.terraform.tfstate"                               # Specify filename mentioned in blob container above.

trigger:
  - none

pool:
  vmImage: "ubuntu-latest"

stages:

  # Performs Terraform Init, Validate, and Plan.
  - stage: applicationinsights
    jobs:
    - job: validate
      continueOnError: false
      steps:

        # Powershell task to enable Storage Account public access so .tfstate file can be opened and modified.
        - task: AzurePowerShell@5
          displayName: 'Temporarily Allow Public Network Access to Terraform Backend Storage Account'
          inputs:
            azureSubscription: '$(serviceConnectionName)'
            ScriptType: 'InlineScript'
            Inline: |
              # Enable Public Network Access
              Set-AzStorageAccount -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -PublicNetworkAccess Enabled
              Update-AzStorageAccountNetworkRuleSet -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -DefaultAction Allow
              Start-Sleep -Seconds 60
            azurePowerShellVersion: 'LatestVersion'

        # Perform Terraform Init
        - task: TerraformTaskV2@2
          displayName: "init"
          inputs:
            provider: "azurerm"
            command: "init"
            workingDirectory: "$(workingDirectory)"
            backendServiceArm: "$(serviceConnectionName)"
            backendAzureRmResourceGroupName: "$(backendResourceGroupName)"
            backendAzureRmStorageAccountName: "$(backendStorageAccountName)"
            backendAzureRmContainerName: "$(backendContainerName)"
            backendAzureRmKey: "$(backendKey)"

        # Perform Terraform Validate    
        - task: TerraformTaskV2@2
          inputs:
            provider: "azurerm"
            command: "validate"
    
    # Perform Terraform Init
    - deployment: plan_terraform
      dependsOn: validate
      continueOnError: false
      environment: "gen-terraform-env"
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: self
              - task: TerraformTaskV2@2
                displayName: "init"
                inputs:
                  provider: "azurerm"
                  command: "init"
                  workingDirectory: "$(workingDirectory)"
                  backendServiceArm: "$(serviceConnectionName)"
                  backendAzureRmResourceGroupName: "$(backendResourceGroupName)"
                  backendAzureRmStorageAccountName: "$(backendStorageAccountName)"
                  backendAzureRmContainerName: "$(backendContainerName)"
                  backendAzureRmKey: "$(backendKey)"

              # Perform Terraform Plan    
              - task: TerraformTaskV2@2
                displayName: "plan"
                inputs:
                  provider: "azurerm"
                  command: "plan"
                  workingDirectory: "$(workingDirectory)"
                  commandOptions: '-var-file="$(varFile)" -var="backendResourceGroupName=$(backendResourceGroupName)" -var="backendStorageAccountName=$(backendStorageAccountName)" -var="backendContainerName=$(backendContainerName)" -var="backendKey=$(backendKey)"'
                  environmentServiceNameAzureRM: "$(serviceConnectionName)"

              # Powershell job to disable Storage Account public access so .tfstate file is secured.     
              - task: AzurePowerShell@5
                displayName: 'Disable Public Network Access to Terraform Backend Storage Account'
                inputs:
                  azureSubscription: '$(serviceConnectionName)'
                  ScriptType: 'InlineScript'
                  Inline: |
                    # Disable Public Network Access
                    Set-AzStorageAccount -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -PublicNetworkAccess Disabled
                  azurePowerShellVersion: 'LatestVersion'  
    
    # Error handling
    - job: catch_failed_plan
      dependsOn: plan_terraform
      condition: failed()
      continueOnError: false
      steps:
        - task: AzurePowerShell@5
          displayName: 'Disable Public Network Access to Terraform Backend Storage Account'
          inputs:
            azureSubscription: '$(serviceConnectionName)'
            ScriptType: 'InlineScript'
            Inline: |
              # Disable Public Network Access
              Set-AzStorageAccount -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -PublicNetworkAccess Disabled
            azurePowerShellVersion: 'LatestVersion'

15. Run plan pipeline script terraform-plan.yaml

Terraform Apply – terraform-apply.yaml

16. Paste the pipeline code in your terraform-apply.yaml file and change the variables that you mentioned created in the earlier steps.

# Terraform APPLY Pipeline configuration

parameters:
- name: environment
  displayName: Environment OneManITArmy
  type: string
  values:
  - dev

# Create if statement to decide which environment Terraform needs to deploy.
variables:
  ${{ if eq(parameters.environment, 'dev') }}:
    serviceConnectionName: "<your service connection name>"                                      # Specify service connection name in Azure DevOps project settings.
    workingDirectory: '$(System.DefaultWorkingDirectory)'                                        # Specify working directory of your Terraform files.
    varFile: "$(System.DefaultWorkingDirectory)/env-tfvars/${{ parameters.environment }}.tfvars"
    backendResourceGroupName: "<your resource group name>"                                       # Specify RG-name that is created via Terraform where Storage account is held.
    backendStorageAccountName: "<your new storage account name>"                                 # Specify name of the Storage account is that created.
    backendContainerName: "<your tfstate blob container name>"                                   # Specify name of blob container in Storage account.
    backendKey: "<your tfstate backendkey name>.terraform.tfstate"                               # Specify filename mentioned in blob container above.


trigger:
  - none

pool:
  vmImage: "ubuntu-latest"

stages:

  # Performs Terraform Init, Validate, and Apply.
  - stage: applicationinsights
    jobs:
    - job: validate
      continueOnError: false
      steps:

        # Powershell task to enable Storage Account public access so .tfstate file can be opened and modified.
        - task: AzurePowerShell@5
          displayName: 'Temporarily Allow Public Network Access to Terraform Backend Storage Account'
          inputs:
            azureSubscription: '$(serviceConnectionName)'
            ScriptType: 'InlineScript'
            Inline: |
              # Enable Public Network Access Terraform Backend Storage Account
              Set-AzStorageAccount -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -PublicNetworkAccess Enabled
              Start-Sleep -Seconds 60
            azurePowerShellVersion: 'LatestVersion'

        # Perform Terraform Init          
        - task: TerraformTaskV2@2
          displayName: "init"
          inputs:
            provider: "azurerm"
            command: "init"
            workingDirectory: "$(workingDirectory)"
            backendServiceArm: "$(serviceConnectionName)"
            backendAzureRmResourceGroupName: "$(backendResourceGroupName)"
            backendAzureRmStorageAccountName: "$(backendStorageAccountName)"
            backendAzureRmContainerName: "$(backendContainerName)"
            backendAzureRmKey: "$(backendKey)"

        # Perform Terraform Validate        
        - task: TerraformTaskV2@2
          inputs:
            provider: "azurerm"
            command: "validate"

    # Perform Terraform Init once more after validating the code.   
    - deployment: apply_terraform
      dependsOn: validate
      continueOnError: false
      environment: "gen-terraform-env"
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: self
              - task: TerraformTaskV2@2
                displayName: "init"
                inputs:
                  provider: "azurerm"
                  command: "init"
                  workingDirectory: "$(workingDirectory)"
                  backendServiceArm: "$(serviceConnectionName)"
                  backendAzureRmResourceGroupName: "$(backendResourceGroupName)"
                  backendAzureRmStorageAccountName: "$(backendStorageAccountName)"
                  backendAzureRmContainerName: "$(backendContainerName)"
                  backendAzureRmKey: "$(backendKey)"

              # Perform Terraform Apply     
              - task: TerraformTaskV2@2
                displayName: "apply"
                inputs:
                  provider: "azurerm"
                  command: "apply"
                  workingDirectory: "$(workingDirectory)"
                  commandOptions: '-var-file="$(varFile)" -var="backendResourceGroupName=$(backendResourceGroupName)" -var="backendStorageAccountName=$(backendStorageAccountName)" -var="backendContainerName=$(backendContainerName)" -var="backendKey=$(backendKey)"'
                  environmentServiceNameAzureRM: "$(serviceConnectionName)"
              
              # Powershell job to disable Storage Account public access so .tfstate file is secured. 
              - task: AzurePowerShell@5
                displayName: 'Disable Public Network Access to Terraform Backend Storage Account'
                inputs:
                  azureSubscription: '$(serviceConnectionName)'
                  ScriptType: 'InlineScript'
                  Inline: |
                    # Disable Public Network Access
                    Set-AzStorageAccount -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -PublicNetworkAccess Disabled
                  azurePowerShellVersion: 'LatestVersion'

    # Error handling                              
    - job: catch_failed_apply
      dependsOn: apply_terraform
      condition: failed()
      continueOnError: false
      steps:
        - task: AzurePowerShell@5
          displayName: 'Disable Public Network Access to Terraform Backend Storage Account'
          inputs:
            azureSubscription: '$(serviceConnectionName)'
            ScriptType: 'InlineScript'
            Inline: |
              # Disable Public Network Access Terraform Backend Storage Account.
              Set-AzStorageAccount -ResourceGroupName "$(backendResourceGroupName)" -Name "$(backendStorageAccountName)" -PublicNetworkAccess Disabled
            azurePowerShellVersion: 'LatestVersion'

17. Run plan pipeline script terraform-apply.yaml

Congratulations! You can now enroll your Terraform code in Azure via Azure DevOps!

The entire code can be downloaded in my repository.