I am writing PowerShell scripts to delete TeamCity work directory on daily basis, the script is working well in ideal condition(when none of the folder in work directory is in use). But when any build is executing in TeamCity and during that time if I execute the script, it shows failure.
#Cleanup work directory of TeamCity's build agent
$Path = "C:\BuildAgent\work"
$exclude = #("*.old", "*directory.map")
Get-ChildItem $Path -Exclude $exclude | Remove-Item -Recurse
Can anyone suggest how can I skip this particular condition without showing any failure so that all the files are deleted except the one which is in use.
There is an undocumented API call that allows you to clean sources on a particular agent.
Loop through all agents and make a POST request to:
http://teamcity/ajax.html?resetSources=<agentID>
Assuming you are using a new version of TeamCity, you will need to be aware of CSRF protection.
Related
In our pipelines building .NET apps, we use SonarScanner for Azure DevOps version 4.23.1, using the MSBuild integration. Sometimes, the tasks leave some mess in the .sonarqube directory inside the pipeline's workspace. We use a PowerShell script to delete the folder. Today, it started failing on an agent in a very strange way. It seems the directory's removal is blocked by a file that cannot even be listed be deleted, moved, nor opened. It happens only on the specific agent, but it is 100% reproducible.
The script:
[string]$sqPath = "$(Agent.BuildDirectory)\.sonarqube\"
Write-Host $sqPath
if (Test-Path -Path $sqPath) {
Write-Host "Path exists!"
#Get-ChildItem $sqPath -Recurse -Force # DEBUG
Remove-Item $sqPath -Recurse -Force
} else {
Write-Host "Path doesn't exist."
}
The output:
F:\Agents\01-V2\_work\104\.sonarqube\
Path exists!
Remove-Item : Cannot remove item F:\Agents\01-V2\_work\104\.sonarqube\out\.sonar\mod37\ucfg2\js\F__Agents_01_V2__work_1
04_s_Cmpny_Prod_Plugins_SitePrototypes_Cmpny_Prod_Plugins_SitePrototypes_Importer_Modules_ModelCharacterGallery_carouse
l_init_js_62_9_FD_45_tryToGetLargeYoutubeThumbnail.ucfg: Could not find a part of the path 'F__Agents_01_V2__work_104_s
_Cmpny_Prod_Plugins_SitePrototypes_Cmpny_Prod_Plugins_SitePrototypes_Importer_Modules_ModelCharacterGallery_carousel_in
it_js_62_9_FD_45_tryToGetLargeYoutubeThumbnail.ucfg'.
At F:\Agents\01-V2\_work\_temp\eac8036c-a258-4f5e-b3c0-16ce3b6a00d9.ps1:9 char:5
+ Remove-Item $sqPath -Recurse -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : WriteError: (F__Agents_01_V2...eThumbnail.ucfg:FileInfo) [Remove-Item], DirectoryNotFoun
dException
+ FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
##[error]PowerShell exited with code '1'.
When I uncomment the Get-ChildItem line, the file mentioned by the error message is not listed anywhereI added -Force and it appeared. is among the listed files, in a hidden directory, yet the Remove-Item line still fails with the same error.
The agents are custom Azure VMs running Windows, managed by the infrastructure department of our company. We have little info on what is going on there and we can use them just through the pipelines. When we send a ticket to the infrastructure admins, it takes ages to get any response and much time is wasted.
Is there a way I can diagnose the issue through the pipeline script? I tried different ways of finding the process that holds the handle to the file but all failed. Most strangely, I found that, unlike *nix systems, Windows does not allow the deletion of a file that is open by a process. So is this an issue with rights? Why is Get-ChildItem silent about the file when Remove-Item fails during its deletion? Why does any operation with the file fail? Historically, we had issues with too-long paths, but this time, it seems there is no PathTooLongException.
I suspect that this issue is/was being caused by Windows Maximum Path Length limitations
As detailed in that article:
In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters. A local path is structured in the following order: drive letter, colon, backslash, name components separated by backslashes, and a terminating null character.
If my counting ability is correct, I see that path length is 265 characters, just above the 260 character limit for a path.
Part of the path looks to me to be randomly generated, so could explain why the issue is not occurring all the time.
When running recent versions of Windows, it's possible to enable NTFS Long Path support which will allow this limit to be bypassed by some applications (namely those that have been built with long path awareness - as far as I can tell PowerShell is one such application)
If you'd like to enable NTFS Long Paths, there's an article you can find here that details the process, and just in case the link gets downed at any point, the key steps are:
Click Windows key and type gpedit.msc, then press the Enter key. This
launches the Local Group Policy Editor.
Navigate to Local Computer Policy > Computer Configuration >
Administrative Templates > System > Filesystem.
Double click Enable NTFS long paths.
Select Enabled, then click OK.
The error messages are misleading, likely because an old PowerShell is used (v1.0, according to the path 😱).
SonarQube somehow managed to create files whose path is longer than MAX_PATH. MAX_PATH is 260 and it includes the trailing NUL character (string terminator). Your path has a length of 261, in this sense, which is just over the limit.
To delete the directory, use this pipeline:
pool:
name: Build
demands: Agent.Name -equals Agent-SKDACLSADOBW01-01-V2
variables:
sq: \\?\F:\Agents\01-V2\_work\104\.sonarqube\
steps:
- checkout: none
- script: rd /s /q $(sq)
It executes the rd (a.k.a. rmdir) command in recursive (/s), quiet (/q) mode on the folder using cmd.exe. This evades many of the idiosyncrasies and bugs of PowerShell. If you need to execute it from within PowerShell, use &cmd.exe /c rd /s /q $(sq). The path needs to be specified as an extended-length path using the \\?\ prefix. Note that when the prefix is used, forward slashes are not converted to backslashes automatically anymore.
In my case, the one-time cleanup described above did not help as the pipeline still produces the files. I modified the script from the question so that it does not have to be modified to run diagnostics and so that it logs a warning and tags the build sonarqube-cleanup each time the cleanup logic is actually used. Based on the tag, I will later be able to find whether the cleanup logic still needs to be there.
if ($env:SYSTEM_DEBUG -eq 'true') {
Write-Host "##[debug]Processes on the agent:"
Get-Process
}
[string]$sqPath = "$(Agent.BuildDirectory)\.sonarqube\"
if (Test-Path -Path $sqPath) {
Write-Host "##vso[task.logissue type=warning]SonarQube folder exists! $sqPath"
Write-Host "##vso[build.addbuildtag]sonarqube-cleanup"
if ($env:SYSTEM_DEBUG -eq 'true') {
Write-Host "##[debug]Contents of the SonarQube folder:"
Get-ChildItem $sqPath -Recurse -Force
}
#Remove-Item $sqPath -Recurse -Force
&cmd.exe /c rd /s /q "\\?\$sqPath"
} else {
Write-Host "SonarQube folder does not exist. $sqPath"
}
I found an interesting GitHub issue about PowerShell support for extended-length paths.
There are many Q & A on Stack Overflow and Super User that detail how files or folders with long paths can be deleted:
Delete directory regardless of 260 char limit
https://superuser.com/q/78434/269404
I am very new to windows / powershell scripting and I am trying atm to get a windows runner on github actions up and running for a java project. For this I need to simulate the following command I have for my linux runner:
find . -path '*/target/surefire-reports/*.xml' | zip -q test-reports.zip -#
So, check multiple target/surefire-reports/ directories of my project, get all XML files and zip them into one archive. This archive will then be uploaded in a later step as a CI artifact.
I read through multiple threads here and on other sites, but couldn't come up with something. I think I know I have to combine Get-ChildItem and Compress-Archive, but couldn't find a way to use these two together with multiple paths as of now.
My "solution" atm looks like this:
run: |
mvn clean install -nsu -B -f project/tests/pom.xml
# TODO
# $xmlReports = Get-ChildItem -Path .\*\target\surefire-reports -Directory |Get-ChildItem -File -Filter *.xml
# $xmlReports |ForEach-Object
TEST_RESULT=${PIPESTATUS[0]}
exit $TEST_RESULT
But as you can see it is unfinished and wouldn't work for sure. The runner runs on windows-latest on Github Actions, so this would be Windows Server 2022 and powershell >v5 at the time of writing this.
I am a bit stuck at the moment and hope someone here could help me out. Thanks in advance.
Sorry it's probably a dumb question: what is the difference between
Remove-Item -recurse -Force -Verbose and Remove-Item -Force -Verbose
it seems if we use -recurse for a folder and subfolders powershell delete file one by one inside the folder. and if we remove the -recurse powershell simply delete the main folder without checking inside the folder. technically isn't script will run faster without -recurse?
tl;dr
Pass both -Recurse and -Force to Remove-Item in order to (more) predictably delete (remove) a given folder, which - barring any permission and timing problems - deletes the folder and any contents it may have.
Caveat: This instantly deletes any contents inside the target folder, and, given that deleted items are not placed in the operating system's recycle bin, can only be recovered, potentially, with specialty software.
it seems if we use -recurse for a folder and subfolders powershell delete file one by one inside the folder.
More accurately, it deletes the target folder's subtree, i.e. it recursively deletes all files and subfolders located in the target folder, including their files and subfolders, recursively, before deleting the target folder itself. In effect, it deletes the target folder and all of its contents, if any, but note the caveats:
re "protected" items, which additionally require -Force, discussed below.
re intermittent failures due to the file-system APIs being asynchronous in older Windows versions, discussed in the bottom section.
In fact, deleting all the contents of a folder before deleting the folder itself is the only way to delete a nonempty folder, technically: the file-system APIs do not offer deletion of a nonempty folder as a single operation.
Because inadvertently deleting an entire subfolder tree can have disastrous consequences, as a safety mechanism PowerShell requires you to signal the intent to delete a nonempty folder explicitly - by passing -Recurse.
If you neglect to pass -Recurse and the target folder is nonempty, you get an interactive confirmation prompt - irrespective of whether -Force is specified or not. Choose [A] Yes to All (type a) to delete the folder and all its contents - but see the situational additional need for -Force below.
That said, you do also need -Force in order to (more) predictably remove a nonempty target folder, because -Force makes PowerShell also delete "protected" files and folders, which are hidden files and folders and files that have the ReadOnly and/or System attributes set (on Windows).
If you neglect to pass -Force when you use -Recurse or interactively choose [A] Yes to All in response to the confirmation prompt, the presence of at least one protected item will prevent removal of the target folder as a whole, though unprotected items inside the subtree will get deleted right away.
Each protected item will cause a non-terminating error to be emitted, concluded by a non-terminating error that the target folder cannot be removed, because it isn't empty (yet). Perhaps confusingly, in Windows PowerShell the per-protected-item error messages only talks about "[in]sufficient access rights", even though the real problem in this case isn't one of permissions; the error message has been amended in _PowerShell (Core) 7+ to explicitly mention hidden, system, and readonly items.
if we remove the -recurse powershell simply delete the main folder without checking inside the folder.
No: It follows from the above that you cannot delete a given nonempty folder unless you delete its contents first.
If you attempt that without -Recurse, you'll invariably get the confirmation prompt (or, in non-interactive scenarios, the call will fail outright).
technically isn't script will run faster without -recurse?
It also follows from the above that only an empty folder can be removed without -Recurse without triggering the confirmation prompt.
If you do also specify -Recurse when targeting an empty folder, hypothetically unnecessary work of testing whether child items exist could be performed. In practice, Remove-Item's implementation always performs this test, whether or not you pass -Recurse.
Even with both -Recurse and -Force specified, overall removal may fail:
... due to insufficient file-system permission held by the current user relative to the target folder and its contents.
... intermittently, due to running on Windows versions older than Windows 10 20H2 (I don't know that Windows Server version that corresponds to), because file-system item deletion there was inherently asynchronous(!), resulting in intermittent failure to fully delete a given target folder, namely if deletion of an item inside the folder hadn't completed yet by the time deletion of the folder itself was attempted: see this answer for details and a workaround.
when you use Remove-Item on a dir tree that holds files ... and do NOT use -Recurse, you will get the standard confirmation prompt. so using that parameter makes it run without the delay from the "do you really want to do this?" prompt.
this is one of the reasons that some folks prefer to pipe the output of Get-ChildItem -Recurse to Remove-Item.
I'm using advanced installer for creating msi package I want copy some files and folders after installation completed to "[APPDIR]" (I know I can do this with add files and folder to files and folder section in advanced installer but I don't want to do that because my files and folder are dynamic in each installation in customer machine)
I write an inline PowerShell script like below
> Param( [string] $source, [string] $dest )
$exclude = #('web.config')
> Get-ChildItem $source -Recurse -Exclude $exclude | Copy-Item
> -Destination {Join-Path $dest $_.FullName.Substring($source.length)}
and in the parameter section, I fill like this "[SourceDir]Project", "[APPDIR]Project"
but it doesn't work. Why?
Abbas has since confirmed that the problem was one of command-line (parameter) syntax:
The parameter section - what to pass to the PowerShell script from Advanced Installer - was filled in as:
"[SourceDir]Project", "[APPDIR]Project" # !! WRONG, due to the comma
whereas it should have been:
"[SourceDir]Project" "[APPDIR]Project" # OK: *space-separated* arguments
Calling scripts/functions/cmdlets in PowerShell works as it does in shells, not as in programming languages; that is, you must separate the arguments being passed with spaces.
By contrast, using , between tokens constructs an array that is passed as a single argument.
From PowerShell run Get-Help about_Command_Syntax for more information.
It depends, you need to give more details. What are the execution settings for your PS custom action?
Have you checked the verbose log to see the params are passed correctly?
Your custom action should be scheduled as deferred with no impersonation, so it is executed after the APPDIR folder is created by the setup package and it has all the rights to write in that location.
Also, you should add rollback and uninstall custom actions to cleanup the files, as during an uninstalled or a canceled/failed installation, those resources will not be cleaned up by Windows Installer.
I'm using Chef Kitchen to do a POC. We have a PowerShell script which goes to a network path, sorts the build based on date [Install shield installer], copies to the VM in a folder.
This script works correctly when I run inside the VM manually using Power Shell. When I run using Chef Kitchen I don't get any error, the recipe is run but only the first line.I can see the folder called Build created in the VM.
I did an experiment and added a second line to create another folder, it worked correctly. So the issue is the logic I used to copy and paste the build doesn't seem to work with Chef.
The first line creates a new folder, the second and third sets the source and destination path. The fourth line copies the build from the network path [Sort by date to get the latest] to the VM. And the last one is to rename.
And also is there an efficient way to copy a file from a network path [Sorting by date] to a VM using Chef?
powershell_script 'CopyBuild' do
code <<-EOH
New-Item -ItemType directory -Path C:\Build
$source = gci \\BuildPath\Build | ? { $_.PSIsContainer } | sort CreationTime -Desc | select -f 1
$destination = "C:\Build"
Copy-Item "\\BuildPath\Build\$source\*" $destination
Get-ChildItem $destination -r -i "*app1*.exe" | Rename-Item -NewName {"Test.exe"}
EOH
end
I also tried this, calling the PS script:
cookbook_file 'getbuild.ps1' do
mode '0755'
end
This looks like a windows remoting problem, your powershell needs the right configuration to be allowed to access network shares.
To verify if this is the case: replace the UNC path and just copy a local directory - if this works you can dig further to setup the requirements for windows remoting, see the following links for further help:
Safely running windows automation operations that fail inside winrm or powershell remoting (see chapter No access to network resources)
Knife issue 655
enabling and using credssp
To sum these links up: you can schedule a local task in your PS which does the copy to achieve this, or setup credssp.