I am using the Azure File Copy tasks in DevOps to copy the artifacts from Git to a Blob storage that sits under my subscription:
However the Azure File Copy tasks never completes. The task continues to run. I'm wondering if it's because of permission issue where my subscription can't write to the Blob storage. Is there something I'm doing wrong?
steps:
- task: AzureFileCopy#5
displayName: 'AzureBlob File Copy'
inputs:
SourcePath: '$(System.DefaultWorkingDirectory)/_ADF-MRSG-Tide-Continuous-Integration-Build/ArmTemplates'
azureSubscription: 'BDAP-PRD-PIPELINE-CONN (xxxxxxxx)'
Destination: AzureBlob
storage: mrsgtideprdwedev
ContainerName: 'azure-artifacts'
EDIT
Log information:
2023-01-12T17:01:15.4423974Z ##[section]Starting: AzureBlob File Copy
2023-01-12T17:01:15.4550000Z ==============================================================================
2023-01-12T17:01:15.4550253Z Task : Azure file copy
2023-01-12T17:01:15.4550388Z Description : Copy files to Azure Blob Storage or virtual machines
2023-01-12T17:01:15.4550590Z Version : 5.210.1
2023-01-12T17:01:15.4550713Z Author : Microsoft Corporation
2023-01-12T17:01:15.4550863Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
2023-01-12T17:01:15.4551268Z ==============================================================================
2023-01-12T17:01:20.7123632Z ##[command]Import-Module -Name C:\Modules\az_9.0.1\Az.Accounts\2.10.4\Az.Accounts.psd1 -Global
2023-01-12T17:01:22.8437792Z ##[warning]Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide
2023-01-12T17:01:31.6898708Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2023-01-12T17:01:32.5179394Z ##[command]Clear-AzContext -Scope Process
2023-01-12T17:01:32.5180271Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant xxx -Credential System.Management.Automation.PSCredential -Environment AzureCloud #processScope
2023-01-12T17:01:35.1239900Z ##[command] Set-AzContext -SubscriptionId xxx -TenantId xxx
2023-01-12T17:01:36.0411697Z ##[command]Import-Module -Name C:\Modules\az_9.0.1\Az.Resources\6.3.1\Az.Resources.psd1 -Global
2023-01-12T17:01:39.1422423Z ##[command]Import-Module -Name C:\Modules\az_9.0.1\Az.Storage\5.0.0\Az.Storage.psd1 -Global
2023-01-12T17:01:41.0099745Z ##[command]Import-Module -Name C:\Modules\az_9.0.1\Az.Compute\5.0.0\Az.Compute.psd1 -Global
2023-01-12T17:01:44.7529810Z ##[command]Import-Module -Name C:\Modules\az_9.0.1\Az.Network\5.0.0\Az.Network.psd1 -Global
2023-01-12T17:01:45.5713749Z ##[warning]The names of some imported commands from the module 'Microsoft.Azure.PowerShell.Cmdlets.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2023-01-12T17:01:45.8587032Z ##[warning]The names of some imported commands from the module 'Az.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2023-01-12T17:11:24.8085548Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2023-01-12T17:11:24.8364623Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2023-01-12T17:11:24.9976554Z ##[error]This request is not authorized to perform this operation.
RequestId:0e6ffe85-601e-0047-3da8-2639da000000
Time:2023-01-12T17:11:24.5146395Z
Status: 403 (This request is not authorized to perform this operation.)
ErrorCode: AuthorizationFailure
Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:0e6ffe85-601e-0047-3da8-2639da000000
Time:2023-01-12T17:11:24.5146395Z</Message></Error>
Headers:
x-ms-request-id: 0e6ffe85-601e-0047-3da8-2639da000000
x-ms-client-request-id: 069ac45e-6638-4d55-9ece-39522ed6f5c5
x-ms-error-code: AuthorizationFailure
Content-Length: 246
Content-Type: application/xml
Date: Thu, 12 Jan 2023 17:11:24 GMT
Server: Microsoft-HTTPAPI/2.0
2023-01-12T17:11:25.0733282Z ##[section]Finishing: AzureBlob File Copy
Related
This is kind of continuation of the issue I faced in here
Let me give a backgroud. This is my yaml pipleine:
parameters:
- name: sub_name # name of the subscription; required
type: string
default: false
steps:
- script: echo "Here is subscription name:" ${{ parameters.sub_name }}
- task: AzurePowerShell#5
displayName: 'Launching Main.yml'
inputs:
azurePowerShellVersion: LatestVersion
azureSubscription: My-SPN # This is my almighty Service Principal
ScriptType: 'FilePath'
ScriptPath: '$(System.DefaultWorkingDirectory)/MyPowerShell.ps1'
ScriptArguments: -sub_name ${{ parameters.sub_name }}
and this is my MyPowerShell.ps1 file:
#param ($sub_name)
Get-AzContext -ListAvailable | Where{$_.Name -match $sub_name} | Set-AzContext
$SID=(Get-AzContext).Subscription.id
Write-Output "The active subscription SID is" $SID
No matter what value the $sub_name is given the output of $SID is always the Subscription Id of my service principal - "My-SPN"
How should I set AzContext properly so it changes active subscription?
The same PowerShell script works fine in Azure CLI but not when yaml has got service principal.
I tried to use Set-AzContext -Subscription $sub_name -TenantId 2a1c169e-715a-412b-b526-05da3f8412fa but ended up with following error:
Starting: Launching Main.yml
==============================================================================
Task : Azure PowerShell
Description : Run a PowerShell script within an Azure environment
Version : 5.209.0
Author : Microsoft Corporation
Help : https://aka.ms/azurepowershelltroubleshooting
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a_temp\adfb7562-7db5-4be6-ae08-dca4664e460c.ps1'"
Added TLS 1.2 in session.
Import-Module -Name C:\Modules\az_7.5.0\Az.Accounts\2.9.1\Az.Accounts.psd1 -Global
WARNING: Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the
same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can
use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure
Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found
here: https://aka.ms/azps-migration-guide
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Connect-AzAccount -ServicePrincipal -Tenant 2a1c169e-715a-412b-b526-05da3f8412fa -Credential System.Management.Automation.PSCredential -Environment AzureCloud #processScope
Set-AzContext -SubscriptionId 72245732-XXXXXXX -TenantId 2a1c169e-XXXXXXXX
##[error]Please provide a valid tenant or a valid subscription.
##[error]PowerShell exited with code '1'.
Added TLS 1.2 in session.
Finishing: Launching Main.yml
Please help on how to change an active subscription either in a yaml or in powershell file.
Thanks.
AzurePowerShell task uses Service Principal to authenticate and access Azure resources. Therefore, your service principal needs to have subscription owner or contributor permission.
Like #ZiyangLiu-MSFT mentioned above the service principal I was using hasn't had enough rights on the subscription. Changed it to the one which have had proper permission and this resolved the issue.
I have prepared a release pipeline in Azure Devops.
The pipeline has a powershell task.
This is the inline code.
Install-Module -Name Az -AllowClobber -Force
Install-Module -Name Bicep -Force
Import-Module Az.Accounts -Force
$PSVersionTable
This is the log output:
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile
-NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a_temp\67e051fe-1dd4-4420-b9c0-577a882c1207.ps1'"
[33;1mWARNING: Both Az and AzureRM modules were detected on this
machine. Az and AzureRM modules cannot be imported in the same session
or used in the same script or runbook. If you are running PowerShell
in an environment you control you can use the 'Uninstall-AzureRm'
cmdlet to remove all AzureRm modules from your machine. If you are
running in Azure Automation, take care that none of your runbooks
import both Az and AzureRM modules. More information can be found
here: https://aka.ms/azps-migration-guide[0m
[32;1mAccount SubscriptionName
TenantId Environment[0m [32;1m-------
---------------- -------- -----------[0m
*** xxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx AzureCloud [32;1mName : [0mxxxxxx (xxxxxxxxxxxxxxxxxxxx) -
xxxxxxxxxxxxxxxxxxxx -
[32;1m[0m*** [32;1mAccount : [0m*** [32;1mEnvironment : [0mAzureCloud [32;1mSubscription
: [0mxxxxxxxxxxxxxxxxxxxxxx [32;1mTenant :
[0mxxxxxxxxxxxxxxxxxxxxxx [32;1mTokenCache : [0m
[32;1mVersionProfile : [0m [32;1mExtendedProperties : [0m{}
Two questions:
what's wrong with characters?
what does it means this "Both Az and AzureRM modules were detected on this machine" and hot to fix it?
I searched for something but i'm confused and not able to distinguish between various things I have found.
Thanks in advance for helping me.
This is due to the newer versions of pwsh using ANSI to colour code the terminal.
You can disable it as part of your pipeline task too
More here:
Powershell - disable colored command output
Following this article I set up Pipelines to push a PowerShell module to my Artifacts feed.
I can install the module on my local machine, but I was wondering how I can do the same in Pipelines? It seems adding the NuGet source is an interactive process, so how can Pipelines add the Artifacts feed as a source?
The issue is that I don't want to have any user interaction in a CI environment.
If you are using self-hosted agent, you need to configure the folder module permission, self-hosted agent run the cmd via service account instead of personal account.
If you are using hosted agent, add the task power shell and enter below script to install the module.
$patToken = "$(pat)" | ConvertTo-SecureString -AsPlainText -Force
$credsAzureDevopsServices = New-Object System.Management.Automation.PSCredential("xxx", $patToken)
Register-PSRepository -Name "PowershellAzureDevopsServices" -SourceLocation "https://pkgs.dev.azure.com/{Org name}/{project name}/_packaging/{feed name}/nuget/v2" -PublishLocation "https://pkgs.dev.azure.com/{Org name}/{project name}/_packaging/{feed name}/nuget/v2" -InstallationPolicy Trusted -Credential $credsAzureDevopsServices
Get-PSRepository
Find-Module -Repository PowershellAzureDevopsServices -Credential $credsAzureDevopsServices
Install-Module -Name Get-Hello -Repository PowershellAzureDevopsServices -Credential $credsAzureDevopsServices
Get-Module -ListAvailable Get-Hello
Result:
Update1
We need to enter the code during the registration of the power supply enclosure repository, the method is Register-PSRepository
This is a certification issue, If we change the authentication method, maybe we don’t need to enter the code.
In addition, We could also install the module via the cmd Install-Module Get-Hello -Scope CurrentUser -Force
Sorry for the very long post, but I am really stuck and hope someone can help.
I've been back and forth so many times and I am hitting many issues with everything that has to do with PowerShell repositories and Azure DevOps agents.
The end goal is to have the latest versions of some PowerShell modules installed as part of a pipeline.
I write various PowerShell modules, package them as NuGets and push them to different repositories (Azure DevOps artifacts, SonaType Nexus OSS)
I then need these modules installed as part of other pipelines.
As there is no built-in way in Azure DevOps to handle PowerShell repositories and import modules, I wrote a script that takes the repository location, name and credentials as parameters, verifies it is registered and installs the module.
When I run this script on ANY machine, it works perfectly
When this script is a PowerShell task on any pipeline - it has various failures, always with cmdlets from PackageManagement
I thought this is because the agent is running it with -NoProfile, but it works for me when I run the script exactly how the agent runs it - "powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:.....'"
I also tried running a cmd task and calling PowerShell to run the script but had the exact same results
The problems I am hitting are:
Get-PSRepository returns NOTHING. Not even PSGallery
When I try to register a repository (using either Register-PSRepository or Register-PackageSource) it throws an error:
PackageManagement\Register-PackageSource : The property 'Values' cannot be found on this object. Verify that the property exists.
as part of my script, I am running these cmdlets to make sure all the required modules are there:
$webclient=New-Object System.Net.WebClient; $webclient.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Install-PackageProvider -Name NuGet -RequiredVersion 2.8.5.208 -Force -Confirm:$false -Verbose; Install-Module PowerShellGet -RequiredVersion 2.2.4 -SkipPublisherCheck -Verbose -Force;
Another error that comes up is this:
PackageManagement\Get-PackageSource : Unable to find repository 'PSGallery'. Use Get-PSRepository to see all available repositories.
I spent hours on this.
What is the correct way to install PS modules from a 3rd party repository (NuGet-based)
Thanks
I noticed that you have push your nuget package to Azure Artifacts.
You could add Powershell tasks to the pipeline and run the following scripts:
Register-PSRepository:
$patToken = "PAT" | ConvertTo-SecureString -AsPlainText -Force
$credsAzureDevopsServices = New-Object System.Management.Automation.PSCredential("email address", $patToken)
Register-PSRepository -Name "PowershellAzureDevopsServices" -SourceLocation "https://pkgs.dev.azure.com/<org_name>/<project_name>/_packaging/<feed_name>/nuget/v2" -PublishLocation "https://pkgs.dev.azure.com/<org_name>/<project_name>/_packaging/<feed_name>/nuget/v2" -InstallationPolicy Trusted -Credential $credsAzureDevopsServices
Then you could register successfully.
Note: I suggest that you could create a Project Scope feed. Or you may get some issues.
Then you could run the following scripts to install the module.
Find-Module -Repository PowershellAzureDevopsServices
Install-Module -Name Get-Hello -Repository PowershellAzureDevopsServices
For more detailed information, you could refer to this Guidance Document.
I have problem with last task, assign policy:
Module AzureRM which contains Resources namespace (with Get-AzureRmPolicyDefinition) should be already included, right?
https://github.com/Azure/azure-powershell/blob/preview/src/ResourceManager/Resources/Commands.Resources/help/Get-AzureRmPolicyDefinition.md
So why I have this error message?
The term 'Get-AzureRmPolicyDefinition' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
When I add to PowerShell task line:
Install-PackageProvider -Name NuGet -Force -Scope CurrentUser
Install-Module -Name Az.Accounts -RequiredVersion 1.6.2 -Force -Scope CurrentUser -AllowClobber
Install-Module -Name AzureRM.Resources -RequiredVersion 6.7.3 -Force -Scope CurrentUser -AllowClobber
Then I will at the end get:
Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.Commands.ResourceManager.Common, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
How to deal with that problem?
2020-01-23T12:42:46.1108266Z ##[section]Starting: Assign Azure Policy
2020-01-23T12:42:46.1190930Z ==============================================================================
2020-01-23T12:42:46.1191016Z Task : Azure PowerShell
2020-01-23T12:42:46.1191088Z Description : Run a PowerShell script within an Azure environment
2020-01-23T12:42:46.1191160Z Version : 4.159.7
2020-01-23T12:42:46.1191219Z Author : Microsoft Corporation
2020-01-23T12:42:46.1191280Z Help : [Learn more about this task](https://go.microsoft.com/fwlink/?LinkID=613749)
2020-01-23T12:42:46.1191368Z ==============================================================================
2020-01-23T12:42:48.5385840Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Accounts\1.6.4\Az.Accounts.psd1 -Global
2020-01-23T12:42:50.3192937Z ##[warning]Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide
2020-01-23T12:42:55.2184342Z ##[command]Clear-AzContext -Scope Process
2020-01-23T12:42:56.2685211Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2020-01-23T12:42:56.8285933Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -Environment AzureCloud
2020-01-23T12:42:57.9191448Z ##[command] Set-AzContext -SubscriptionId 78afced4-1c58-4e66-8242-c042890d34c3 -TenantId ***
2020-01-23T12:42:58.5652039Z ##[command]& 'D:\a\_temp\743dfd85-b908-48ee-9a00-e0ee97b44c8a.ps1'
2020-01-23T12:42:59.9241324Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2020-01-23T12:43:00.1484807Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2020-01-23T12:43:00.6509257Z ##[error]Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.Commands.ResourceManager.Common, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
2020-01-23T12:43:00.6856188Z ##[section]Finishing: Assign Azure Policy
The warning in your log has display the issue caused:
Both Az and AzureRM modules were detected on this machine. Az and
AzureRM modules cannot be imported in the same session or used in the
same script or runbook. If you are running PowerShell in an
environment you control you can use the 'Uninstall-AzureRm' cmdlet to
remove all AzureRm modules from your machine....
Not sure what is your scripts in your previous tasks. But based on this error message, what I can sure is it installed the AzureRm module into the workspace. And as a result, AzureRM module has conflict with Az module in later steps.
Az module is the new module which we develope and used to replace the AzureRm module. But when you install the AzureRm and Az module into one workspace, it easily cause conflict which will lead the error.
There's no error on your scripts shared, so just need run Uninstall-AzureRm before your scripts to avoid this cmdlet conflicts.