Terraform backend cannot connect to storage account - azure-devops

Question

This my from terraform.tf
terraform {
backend "azurerm" {
resource_group_name = "tstate"
storage_account_name = "strorageaccount1"
container_name = "terraform.tfstate"
access_key = "asdfg45454..."
}
}
This fails when my storage account is not in "all networks". My settings of storage account network is given below. Blob storage private or public it works so no problem there. But "all networks" must be enabled for it to work. How can I make it work with "all networks" disabled? The error I get is as follows:
Error: Failed to get existing workspaces: storage: service returned
error: StatusCode=403, ErrorCode=AuthorizationFailure,
ErrorMessage=This request is not authorized to perform this operation.
There is no IP or Vnet needed as Azure default agent is running the devops pipeline. And the SPN has owner access on subscription. What am I missing?

Answer 1

Well, you explicitly forbid almost any service (or server) to access your storage account. with the exception of "trusted Microsoft services". However, your Azure DevOps Build Agent does not fall under that category.
So, you need to whitelist your build agent first. There are two ways you can do this:
Use a self-hosted agent that you run inside a VNET. Then allow access from that VNET in your firewall rules of your storage account
If you want to stick with managed build agents: Run a AZ CLI or Azure Powershell script first, that does fetch the public IP of your build agent (https://api.ipify.org) and add that to your firewall. After terraform finished, have another script that removes that IP exception again.