Issues with PowerShell Repositories on Azure DevOps Agents - powershell

Sorry for the very long post, but I am really stuck and hope someone can help.
I've been back and forth so many times and I am hitting many issues with everything that has to do with PowerShell repositories and Azure DevOps agents.
The end goal is to have the latest versions of some PowerShell modules installed as part of a pipeline.
I write various PowerShell modules, package them as NuGets and push them to different repositories (Azure DevOps artifacts, SonaType Nexus OSS)
I then need these modules installed as part of other pipelines.
As there is no built-in way in Azure DevOps to handle PowerShell repositories and import modules, I wrote a script that takes the repository location, name and credentials as parameters, verifies it is registered and installs the module.
When I run this script on ANY machine, it works perfectly
When this script is a PowerShell task on any pipeline - it has various failures, always with cmdlets from PackageManagement
I thought this is because the agent is running it with -NoProfile, but it works for me when I run the script exactly how the agent runs it - "powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:.....'"
I also tried running a cmd task and calling PowerShell to run the script but had the exact same results
The problems I am hitting are:
Get-PSRepository returns NOTHING. Not even PSGallery
When I try to register a repository (using either Register-PSRepository or Register-PackageSource) it throws an error:
PackageManagement\Register-PackageSource : The property 'Values' cannot be found on this object. Verify that the property exists.
as part of my script, I am running these cmdlets to make sure all the required modules are there:
$webclient=New-Object System.Net.WebClient; $webclient.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Install-PackageProvider -Name NuGet -RequiredVersion 2.8.5.208 -Force -Confirm:$false -Verbose; Install-Module PowerShellGet -RequiredVersion 2.2.4 -SkipPublisherCheck -Verbose -Force;
Another error that comes up is this:
PackageManagement\Get-PackageSource : Unable to find repository 'PSGallery'. Use Get-PSRepository to see all available repositories.
I spent hours on this.
What is the correct way to install PS modules from a 3rd party repository (NuGet-based)
Thanks

I noticed that you have push your nuget package to Azure Artifacts.
You could add Powershell tasks to the pipeline and run the following scripts:
Register-PSRepository:
$patToken = "PAT" | ConvertTo-SecureString -AsPlainText -Force
$credsAzureDevopsServices = New-Object System.Management.Automation.PSCredential("email address", $patToken)
Register-PSRepository -Name "PowershellAzureDevopsServices" -SourceLocation "https://pkgs.dev.azure.com/<org_name>/<project_name>/_packaging/<feed_name>/nuget/v2" -PublishLocation "https://pkgs.dev.azure.com/<org_name>/<project_name>/_packaging/<feed_name>/nuget/v2" -InstallationPolicy Trusted -Credential $credsAzureDevopsServices
Then you could register successfully.
Note: I suggest that you could create a Project Scope feed. Or you may get some issues.
Then you could run the following scripts to install the module.
Find-Module -Repository PowershellAzureDevopsServices
Install-Module -Name Get-Hello -Repository PowershellAzureDevopsServices
For more detailed information, you could refer to this Guidance Document.

Related

Azure Devops powershell release pipeline strange chars

I have prepared a release pipeline in Azure Devops.
The pipeline has a powershell task.
This is the inline code.
Install-Module -Name Az -AllowClobber -Force
Install-Module -Name Bicep -Force
Import-Module Az.Accounts -Force
$PSVersionTable
This is the log output:
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile
-NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a_temp\67e051fe-1dd4-4420-b9c0-577a882c1207.ps1'"
[33;1mWARNING: Both Az and AzureRM modules were detected on this
machine. Az and AzureRM modules cannot be imported in the same session
or used in the same script or runbook. If you are running PowerShell
in an environment you control you can use the 'Uninstall-AzureRm'
cmdlet to remove all AzureRm modules from your machine. If you are
running in Azure Automation, take care that none of your runbooks
import both Az and AzureRM modules. More information can be found
here: https://aka.ms/azps-migration-guide[0m
[32;1mAccount SubscriptionName
TenantId Environment[0m [32;1m-------
---------------- -------- -----------[0m
*** xxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx AzureCloud [32;1mName : [0mxxxxxx (xxxxxxxxxxxxxxxxxxxx) -
xxxxxxxxxxxxxxxxxxxx -
[32;1m[0m*** [32;1mAccount : [0m*** [32;1mEnvironment : [0mAzureCloud [32;1mSubscription
: [0mxxxxxxxxxxxxxxxxxxxxxx [32;1mTenant :
[0mxxxxxxxxxxxxxxxxxxxxxx [32;1mTokenCache : [0m
[32;1mVersionProfile : [0m [32;1mExtendedProperties : [0m{}
Two questions:
what's wrong with characters?
what does it means this "Both Az and AzureRM modules were detected on this machine" and hot to fix it?
I searched for something but i'm confused and not able to distinguish between various things I have found.
Thanks in advance for helping me.
This is due to the newer versions of pwsh using ANSI to colour code the terminal.
You can disable it as part of your pipeline task too
More here:
Powershell - disable colored command output

How to install PowerShell module pushed to Azure Artifacts in Pipelines?

Following this article I set up Pipelines to push a PowerShell module to my Artifacts feed.
I can install the module on my local machine, but I was wondering how I can do the same in Pipelines? It seems adding the NuGet source is an interactive process, so how can Pipelines add the Artifacts feed as a source?
The issue is that I don't want to have any user interaction in a CI environment.
If you are using self-hosted agent, you need to configure the folder module permission, self-hosted agent run the cmd via service account instead of personal account.
If you are using hosted agent, add the task power shell and enter below script to install the module.
$patToken = "$(pat)" | ConvertTo-SecureString -AsPlainText -Force
$credsAzureDevopsServices = New-Object System.Management.Automation.PSCredential("xxx", $patToken)
Register-PSRepository -Name "PowershellAzureDevopsServices" -SourceLocation "https://pkgs.dev.azure.com/{Org name}/{project name}/_packaging/{feed name}/nuget/v2" -PublishLocation "https://pkgs.dev.azure.com/{Org name}/{project name}/_packaging/{feed name}/nuget/v2" -InstallationPolicy Trusted -Credential $credsAzureDevopsServices
Get-PSRepository
Find-Module -Repository PowershellAzureDevopsServices -Credential $credsAzureDevopsServices
Install-Module -Name Get-Hello -Repository PowershellAzureDevopsServices -Credential $credsAzureDevopsServices
Get-Module -ListAvailable Get-Hello
Result:
Update1
We need to enter the code during the registration of the power supply enclosure repository, the method is Register-PSRepository
This is a certification issue, If we change the authentication method, maybe we don’t need to enter the code.
In addition, We could also install the module via the cmd Install-Module Get-Hello -Scope CurrentUser -Force

Microsoft.IdentityModel.Clients.ActiveDirectory not loading on Azure DevOps PowerShell Release Pipeline

Recently I experienced some issue with Azure DevOps PowerShell when attempting to create a ClientCredential and or ClientAssertion. I have the following code which was working on the past for creating a ClientCredential based on the following variables:
TenantId
ClientID (SPN)
Password (SPN Password)
$ResourceUrl = "https://database.windows.net/"
$AuthorityUrl = "https://login.microsoftonline.com/$($TenantId)"
$objClientCredential = [Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential]::new($ClientId, $Password)
$objAuthenticationContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]::new($AuthorityUrl)
$objAuthenticationResult = $objAuthenticationContext.AcquireTokenAsync($ResourceUrl, $objClientCredential)
but recently this code stop working. It seems that the AzureAD module is not loading correctly. This only happens on Azure DevOps Powershell, on my machine it works fine (I am using PowerShell 7.1)
So far so now I attempted the following:
Run the code as shown above
**Breaks on: **$objClientCredential = [Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential]::new($ClientId, $Password)
Error: Unable to find type [Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential]
Install AzureAD before running the code
Install-Module -Name AzureAD -Force
Import-Module -Name AzureAD
**Breaks on: **$objClientCredential = [Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential]::new($ClientId, $Password)
Error: Unable to find type [Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential]
Import the dll Microsoft.IdentityModel.Clients.ActiveDirectory.dll from my machine
Add-Type -Path ".\libraries\Microsoft.IdentityModel.Clients.ActiveDirectory.dll
Doesn't break, however ClientCredential is not created it is returned as null
Have anyone experienced a similar issue? Do you know how should I drive this?
If you run the scripts on local machine. You can check where the assembly is installed and manually import the .dll file. I tested on my local machine. It works fine when i just import the azureAd module:
Install-Module AzureAD -Force
Import-Module -Name AzureAD
If you are run the script in azure pipeline. When you install AzureAD module using Install-Module -Name AzureAD -Force in azure powershell task . You can see from the build log that AzureAD module is installed in folder C:\Program Files\WindowsPowerShell\Modules\AzureAD\2.0.2.128:
And from the log, we can see assembly Microsoft.IdentityModel.Clients.ActiveDirectory.dll doesnot get loaded automatically.
So you can manually load it from module Azure AD installation folder: See below:
Install-Module AzureAD -Force
Import-Module -Name AzureAD
Add-Type -Path "C:\Program Files\WindowsPowerShell\Modules\AzureAD\2.0.2.128\Microsoft.IdentityModel.Clients.ActiveDirectory.dll"
$TenantId= "..."
$ClientID ="..."
$Password = "..."
$ResourceUrl = "https://database.windows.net/"
$AuthorityUrl = "https://login.microsoftonline.com/$($TenantId)"
...
$objAuthenticationResult = $objAuthenticationContext.AcquireTokenAsync($ResourceUrl, $objClientCredential).Result
See below result.

The term 'Get-AzureRmPolicyDefinition' is not recognized as the name of a cmdlet, function, script file, or operable program

I have problem with last task, assign policy:
Module AzureRM which contains Resources namespace (with Get-AzureRmPolicyDefinition) should be already included, right?
https://github.com/Azure/azure-powershell/blob/preview/src/ResourceManager/Resources/Commands.Resources/help/Get-AzureRmPolicyDefinition.md
So why I have this error message?
The term 'Get-AzureRmPolicyDefinition' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
When I add to PowerShell task line:
Install-PackageProvider -Name NuGet -Force -Scope CurrentUser
Install-Module -Name Az.Accounts -RequiredVersion 1.6.2 -Force -Scope CurrentUser -AllowClobber
Install-Module -Name AzureRM.Resources -RequiredVersion 6.7.3 -Force -Scope CurrentUser -AllowClobber
Then I will at the end get:
Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.Commands.ResourceManager.Common, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
How to deal with that problem?
2020-01-23T12:42:46.1108266Z ##[section]Starting: Assign Azure Policy
2020-01-23T12:42:46.1190930Z ==============================================================================
2020-01-23T12:42:46.1191016Z Task : Azure PowerShell
2020-01-23T12:42:46.1191088Z Description : Run a PowerShell script within an Azure environment
2020-01-23T12:42:46.1191160Z Version : 4.159.7
2020-01-23T12:42:46.1191219Z Author : Microsoft Corporation
2020-01-23T12:42:46.1191280Z Help : [Learn more about this task](https://go.microsoft.com/fwlink/?LinkID=613749)
2020-01-23T12:42:46.1191368Z ==============================================================================
2020-01-23T12:42:48.5385840Z ##[command]Import-Module -Name C:\Modules\az_3.1.0\Az.Accounts\1.6.4\Az.Accounts.psd1 -Global
2020-01-23T12:42:50.3192937Z ##[warning]Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide
2020-01-23T12:42:55.2184342Z ##[command]Clear-AzContext -Scope Process
2020-01-23T12:42:56.2685211Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2020-01-23T12:42:56.8285933Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -Environment AzureCloud
2020-01-23T12:42:57.9191448Z ##[command] Set-AzContext -SubscriptionId 78afced4-1c58-4e66-8242-c042890d34c3 -TenantId ***
2020-01-23T12:42:58.5652039Z ##[command]& 'D:\a\_temp\743dfd85-b908-48ee-9a00-e0ee97b44c8a.ps1'
2020-01-23T12:42:59.9241324Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2020-01-23T12:43:00.1484807Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2020-01-23T12:43:00.6509257Z ##[error]Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.Commands.ResourceManager.Common, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
2020-01-23T12:43:00.6856188Z ##[section]Finishing: Assign Azure Policy
The warning in your log has display the issue caused:
Both Az and AzureRM modules were detected on this machine. Az and
AzureRM modules cannot be imported in the same session or used in the
same script or runbook. If you are running PowerShell in an
environment you control you can use the 'Uninstall-AzureRm' cmdlet to
remove all AzureRm modules from your machine....
Not sure what is your scripts in your previous tasks. But based on this error message, what I can sure is it installed the AzureRm module into the workspace. And as a result, AzureRM module has conflict with Az module in later steps.
Az module is the new module which we develope and used to replace the AzureRm module. But when you install the AzureRm and Az module into one workspace, it easily cause conflict which will lead the error.
There's no error on your scripts shared, so just need run Uninstall-AzureRm before your scripts to avoid this cmdlet conflicts.

Upgrade AzureRM Powershell on Hosted 2017 Agent (VSTS - Visual Studio Team Services)

I am using release management through Visual Studio Teams Services (online). We use Hosted build Agents and I really want to avoid the overhead of managing custom agents.
One item I do need is the AzureRM PowerShell module. Versions up to 5.1.1 are available on the agent but I need 6.0.0.
What I would like to do is use a step in my release process (PowerShell) to aquire version 6.0.0 and use thart instead, however I cant quite get it to work. I have tried a few approaches that have all come unstuck, the current one is:
Write-Output "------------------ Install package provider ------------------"
Find-PackageProvider -Name "NuGet" | Install-PackageProvider -Scope CurrentUser -Force
Write-Output "------------------ Remove Modules ------------------"
Get-Module -ListAvailable | Where-Object {$_.Name -like 'AzureRM*'} | Remove-Module
Write-Output "------------------ Install the AzureRM version we want - 6.0.1! ------------------"
Install-Package AzureRM -RequiredVersion 6.0.1 -Scope CurrentUser -Force
Write-Output "------------------ Import AzureRM 6.0.1 ------------------"
Import-Module AzureRM -RequiredVersion 6.0.1
This all works fine (i.e. does not crash...) but then when I try and use one of the 6.0.1 cmdlets I get an error.
Get-AzureRmADGroup : The Azure PowerShell session has not been
properly initialized. Please import the module and try again.
Any idea of where I am going wrong or alternate strategies I can use to deploy AzureRM 6.0.1 and use it on a hosted agent?
Thanks to Murray for the initial point in the right direction, to show what I was hoping to do wasn't impossible!
I initially tried to do this within the Azure PowerShell task and got pretty far but hit a dead end with AzureRm.Profile as you cannot unload an old version.
The trick was to understanding how the AzureRM VSTS task does it's dependency setup, it effectively takes the "Azure Powershell Version" string in the VSTS UI and uses it to define an additional search path in the PSModules environmental variable i.e. C:\Modules\azurerm_5.1.1.
It will look in that directory, before searching the user profile, then the global modules path.
As soon as it finds the module it performs the Azure login, which will hamper any hope of removing the module after.
So if you instead use a plain powershell task i.e. one that AzureRM isn't loaded into (as Murray also concluded):
Install-PackageProvider -Name NuGet -Force -Scope CurrentUser
Install-Module -Name AzureRM -RequiredVersion 6.2.1 -Force -Scope CurrentUser -AllowClobber
Notably install-module won't be installing to c:\modules like the vsts image generation project.
I seemed to need AllowClobber to get around problems overriding old powershell versions when experimenting, but I suspect I don't need that anymore.
The elegant solution kicks in when next using the Azure PowerShell script.
The preferred powershell version field filled in with 6.2.1 will add C:\Modules\azurerm_6.2.1 to the PSModules path. This doesn't exist, but thankfully because PSModules still includes the user specific modules path, so our 6.2.1 is loaded by itself!
Luckily the AzureRM.Profile from 5.1.1 is forwards compatible enough that the service principal login performed by the Azure Powershell task still works.
Running
Get-Module AzureRm
Get-AzureRmContext
Will output the versions you are hoping for:
Notably if it weren't able to login, I think System.AccessToken could be used (if the option is switched on in the agent phase level).
Diagnostic Techniques if the workaround needs tweaking:
Things that helped greatly, reading through the code that loads in AzureRM in the task:
https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/AzurePowerShellV3/AzurePowerShell.ps1#L80
https://github.com/Microsoft/vsts-tasks/blob/0703b8869041d64db994934bde97de167787ed2e/Tasks/Common/VstsAzureHelpers_/ImportFunctions.ps1
https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/AzurePowerShellV3/Utility.ps1#L18
And also how the VSTS image is generated:
https://github.com/Microsoft/vsts-image-generation/blob/2f57db26dc30ae0f257a3415d26eaa8eea0febf9/images/win/scripts/Installers/Install-AzureModules.ps1
Enabing System.Debug = true as a environment release variable.
Then using VSCode's Log File Highlighter plugin
I'd encourage anyone who's interested to vote in: https://github.com/Microsoft/vsts-image-generation/issues/149
As the AzureRM version at the time of writing is waaaay out of date on the VSTS Hosted 2017 agent.
Unfortunately I can't submit a PR to upgrade it, as it's pulled in via a zip file hosted privately.
I finally figured it out - adding an answer for anyone else that suffers the same.
The key is to login after the AzureRM module is upgraded.
PowerShell code:
Write-Output "------------------ Start: Upgrade AzureRM on build host ------------------"
Write-Output "- - - - - Install package provider"
Install-PackageProvider -Name NuGet -Force -Scope CurrentUser
Write-Output "- - - - - List Modules Before"
Get-Module -ListAvailable| where {$_.Name -Like “*AzureRM*”} | Select Name, Version
Write-Output "- - - - - Remove alll existing AzureRM Modules"
Get-Module -ListAvailable | Where-Object {$_.Name -like '*AzureRM*'} | Remove-Module -Force
Write-Output "- - - - - Install AzureRM 6.0.1"
Install-Module -Name AzureRM -RequiredVersion 6.0.1 -Force -Scope CurrentUser
Write-Output "- - - - - Import AzureRM 6.0.1"
Import-Module AzureRM -Force -Verbose -Scope Local
Write-Output "- - - - - List Modules After"
Get-Module -ListAvailable| where {$_.Name -Like “*AzureRM*”} | Select Name, Version
Write-Output "------------------ End: Upgrade AzureRM on build host ------------------"
Write-Output "------------------ Start: LoginToAzure ------------------"
$SecurePassword = ConvertTo-SecureString $AdminPassword -AsPlainText -Force
$AdminCredential = New-Object System.Management.Automation.PSCredential ($AdminUserEmailAddress, $SecurePassword)
Login-AzureRmAccount -Credential $AdminCredential
Get-AzureRmSubscription –SubscriptionId $SubscriptionId | Select-AzureRmSubscription
Write-Output "------------------ End: LoginToAzure ------------------"