I have a direct delete function (deletes the files direct) implemented in my script, and a trashbin delete function (moving it to the trashbin first).
The problem is that the trashbin delete doesn't work. I've tried this suggested post already but it doesn't seem to work.
My full script :
## Top of the script
param(
[Parameter(Mandatory=$true)]
[ValidateRange(0,99999)]
[int]$minutes,
[Parameter(Mandatory=$true)]
[ValidateScript({Test-Path $_})]
[string]$maplocation,
[Parameter(Mandatory=$true)]
[ValidateSet("Direct", "TrashBin")]
[string]$consequence
)
## error notifications
## Variables
$file = Get-ChildItem -Path $maplocation | Get-Date
$time = Get-Date
$minutesconvert = (New-Timespan -Start $file -End $time).TotalMinutes
foreach ($file in $files)
{
if ($minutes -lt $minutesconvert -and $consequence -eq "direct")
{
Write-Verbose "File Found $file" -Verbose
Write-Verbose "Deleting $file" -Verbose
Remove-Item $file.FullName
}
elseif ($minutes -lt $minutesconvert -and $consequence -eq "trashbin")
{
Add-Type -AssemblyName Microsoft.VisualBasic
Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile($maplocation, 'OnlyErrorDialogs', 'SendToRecycleBin')
}
else
{
Write-Verbose -message "txt" -verbose
}
}
Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile($maplocation, 'OnlyErrorDialogs', 'SendToRecycleBin')
Error code in PowerShell console:
New-TimeSpan : Cannot convert 'System.Object[]' to the type 'System.DateTime' required
by parameter 'Start'. The method is not supported.
At C:\Users\david\Desktop\nieuw.ps1:21 char:39
+ $minutesconvert = (New-TimeSpan -Start <<<< $file -End $time).TotalMinutes
+ CategoryInfo : InvalidArgument: (:) [New-TimeSpan], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgument,Microsoft.PowerShell.Commands.NewTimeSpanCommand
This is your culprit:
$file = Get-ChildItem -Path $maplocation | Get-Date
The above statement will give you the current date and time for each file and folder in $maplocation. If $maplocation is not a single file the result is an array, which New-TimeSpan is not prepared to handle. The procedure is also very unlikely to be what you actually intended. You probably want the time difference between the last modification (creation) date of $maplocation (or its contents?). Besides, rather than calculating a timespan it's better to subtract the number of minutes from the current timestamp and use that as a reference date.
Also, depending on what you want to do in case $maplocation is a folder, you may need to process the item differently:
$maplocation is a folder and you want to delete the folder and everything in it, or $maplocation is a single file:
$maxAge = (Get-Date).AddMinutes(-$minutes)
$file = Get-Item $maplocation
if ($file.LastWriteTime -lt $maxAge) {
switch ($consequence) {
'direct' { ... }
'trashbin' { ... }
}
}
$maplocation is a folder and you want to delete only items from it that are older than the reference date:
$maxAge = (Get-Date).AddMinutes(-$minutes)
$files = Get-ChildItem $maplocation -Recurse
foreach ($file in $files) {
if ($file.LastWriteTime -lt $maxAge) {
switch ($consequence) {
'direct' { ... }
'trashbin' { ... }
}
}
}
Since the sample code from your question is incomplete further adjustments may be required.
Related
I've got this script that connects to Sharepoint Online, indexes all the files and folders, downloads them all in a systematic fashion and churns out a .csv with the name of file, folders, size, path, etc.
For various reasons I've ended up in a situation where I've got all the data, but the metadata is corrupted (the .csv file aforementioned).
Unfortunately re running the whole script just for that isn't really an option, as that would require around 90 hours.
I've been trying to break the code down in order to remove the "download files" functions and just keep the part that generates the .csv, but so far without luck.
I've found the Function that seem to be in charge of it (WriteLog), but I'm struggling to separate it from the rest.
P.S. The code is not mine, I've inherited it from a developer I haven't got access to (unfortunately)
Please find the code below:
param(
[Parameter(Mandatory = $true)]
[string]$srcUrl,
[Parameter(Mandatory = $true)]
[string]$username,
[Parameter(Mandatory = $false,HelpMessage = "From Date: (dd/mm/yyyy)")]
[string]$fromDate,
[Parameter(Mandatory = $false,HelpMessage = "To Date: (dd/mm/yyyy)")]
[string]$toDate,
[Parameter(Mandatory = $true)]
[string]$folderPath,
[Parameter(Mandatory = $true)]
[string]$csvPath
) #end param
cls
#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\SharePoint Online Management Shell\Microsoft.Online.SharePoint.PowerShell\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\SharePoint Online Management Shell\Microsoft.Online.SharePoint.PowerShell\Microsoft.SharePoint.Client.Runtime.dll"
$global:OutFilePath = -join ($csvPath,"\Documents.csv")
$global:OutFilePathError = -join ($csvPath,"\ErrorLog_GetDocuments.csv")
$header = "Title,Type,Parent,Name,Path,FileSize(bytes),Created,Created by,Modified,Modified by,Matterspace title,Matterspace url"
$srcLibrary = "Documents"
$securePassword = Read-Host -Prompt "Enter your password: " -AsSecureString
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials ($username,$securePassword)
$sUrl = [System.Uri]$srcUrl
$domainUrl = -join ("https://",$sUrl.Host)
function WriteLog
{
param(
[Parameter(Mandatory = $true)] $title,$type,$folderName,$name,$path,$fileSize,$created,$createdby,$modifed,$modifiedby,$matterspacetitle,$materspaceUrl
)
$nowTime = Get-Date -Format "dd-MMM-yy,HH:mm:ss"
$folderName = $folderName.Replace(",","|") ### sometime folder / file name has comma so replace it with something
$name = $name.Replace(",","|")
#$path = $path.Replace(",","|")
$title=[System.String]::Concat("""""""$title""""""")
$type=[System.String]::Concat("""""""$type""""""")
$folderName=[System.String]::Concat("""""""$folderName""""""")
$name=[System.String]::Concat("""""""$name""""""")
$path=[System.String]::Concat("""""""$path""""""")
$fileSize=[System.String]::Concat("""""""$fileSize""""""")
$created=[System.String]::Concat("""""""$created""""""")
$createdby=[System.String]::Concat("""""""$createdby""""""")
$modified=[System.String]::Concat("""""""$modified""""""")
$modifiedby=[System.String]::Concat("""""""$modifiedby""""""")
$matterspacetitle=[System.String]::Concat("""""""$matterspacetitle""""""")
$materspaceUrl=[System.String]::Concat("""""""$materspaceUrl""""""")
$lineContent = "$("$title"),$($type),$($folderName),$($name),$($path),$($fileSize),$($created),$($createdby),$($modified),$($modifiedby),$($matterspacetitle),$($materspaceUrl)"
Add-Content -Path $global:OutFilePath -Value "$lineContent"
}
#Function to get all files of a folder
Function Get-FilesFromFolder([Microsoft.SharePoint.Client.Folder]$Folder,$SubWeb,$MTitle)
{
Write-host -f Yellow "Processing Folder:"$Folder.ServerRelativeUrl
$folderItem = $Folder.ListItemAllFields
#$srcContext.Load($f)
$Ctx.Load($folderItem)
$Ctx.ExecuteQuery()
#Get All Files of the Folder
$Ctx.load($Folder.files)
$Ctx.ExecuteQuery()
$authorEmail = $folderItem["Author"].Title
$editorEmail = $folderItem["Editor"].Title
$filepath = $folderItem["FileDirRef"]
if([string]::IsNullOrEmpty($filepath))
{
$filepath=$Folder.ServerRelativeUrl
}
$created = $folderItem["Created"]
$modified = $folderItem["Modified"]
$title = $folderItem["Title"]
if ([string]::IsNullOrEmpty($title))
{
$title = "Not Specified"
}
#$fileSize = $fItem["File_x0020_Size"]
$fileName = $Folder.Name
#list all files in Folder
write-host $Folder.Name
$splitString=$Folder.ServerRelativeUrl -split('/')
$dirUrl="";
write-host $splitString.Length
$parentUrl=""
For($i=3; $i -le $splitString.Length;$i++)
{
if($splitString[$i] -notcontains('.'))
{
Write-Host $i
Write-Host $splitString[$i]
$dirUrl=-join($dirUrl,"\",$splitString[$i])
$parentUrl=-join($parentUrl,"\",$splitString[$i+1])
}
}
$dirPath = -join ($folderPath,$dirUrl)
WriteLog $title "Folder" $parentUrl.TrimEnd('\') $fileName $filepath 0 $created $authorEmail $modified $editorEmail $MTitle $SubWeb
write-host $dirPath
if (-not (Test-Path -Path $dirPath))
{
New-Item -ItemType directory -Path $dirPath
}
ForEach ($File in $Folder.files)
{
try{
$remarkDetail = ""
$replacedUser = ""
$fItem = $File.ListItemAllFields
#$srcContext.Load($f)
$Ctx.Load($fItem)
$Ctx.ExecuteQuery()
$authorEmail = $fItem["Author"].Email
$editorEmail = $fItem["Editor"].Email
$filepath = $fItem["FileDirRef"]
$fileSizeBytes = $fItem["File_x0020_Size"];
$fileSize = ($fileSizeBytes) / 1MB
$fileName = $fItem["FileLeafRef"]
$title = $fItem["Title"]
$filecreated = $fitem["Created"]
$fileModified = $fitem["Modified"]
$FileUrl = $fItem["FileRef"]
$Fname=$File.Name
if ([string]::IsNullOrEmpty($title))
{
$title = "Not Specified"
}
#$title,$type, $folderName,$name,$path,$fileSize,$created,$createdby,$modifed,$modifiedby,$matterspacetitle,$materspaceUrl
$dateToCompare = Get-Date (Get-Date -Date $fileModified -Format 'dd/MM/yyyy')
#Get the File Name or do something
if (($dateToCompare -ge $startDate -and $dateToCompare -le $endDate) -or ($startDate -eq $null -and $endDate -eq $null))
{
$downloadUrl = -join ($dirPath,$File.Name)
$fromfile = -join ($domainUrl,$FileUrl)
Write-Host "Downloading the file from " $fromfile -ForegroundColor Cyan
try{
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials ($username,$securePassword)
$webclient.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED","f")
$webclient.DownloadFile($fromfile,$downloadUrl)
$webclient.Dispose()
}
catch{
$ErrorMessage=$_.Exception.Message
$ErrorMessage = $ErrorMessage -replace "`t|`n|`r",""
$ErrorMessage = $ErrorMessage -replace " ;|; ",";"
$lineContent = "$($Fname),$($fromfile ),$($ErrorMessage)"
Add-Content -Path $global:OutFilePathError -Value "$lineContent"
Write-Host "Skipping the file and recalling the function" -ForegroundColor Blue
}
WriteLog $title "File" $Folder.Name $fileName $FileUrl $fileSize $created $authorEmail $modified $editorEmail $MTitle $SubWeb
Write-host -f Magenta $File.Name
}
else
{
Write-Host "Skipping the matterspace :" $title " as the matterspace was not in the date range" -ForegroundColor Blue
}
}
catch{
$ErrorMessage=$_.Exception.Message
$ErrorMessage = $ErrorMessage -replace "`t|`n|`r",""
$ErrorMessage = $ErrorMessage -replace " ;|; ",";"
$lineContent = "$($Fname),$($fromfile ),$($ErrorMessage)"
Add-Content -Path $global:OutFilePathError -Value "$lineContent"
}
}
#Recursively Call the function to get files of all folders
$Ctx.load($Folder.Folders)
$Ctx.ExecuteQuery()
#Exclude "Forms" system folder and iterate through each folder
ForEach($SubFolder in $Folder.Folders | Where {$_.Name -ne "Forms"})
{
Get-FilesFromFolder -Folder $SubFolder -SubWeb $SubWeb -Mtitle $MTitle
}
}
Function Get-SPODocLibraryFiles()
{
param
(
[Parameter(Mandatory=$true)] [string] $SiteURL,
[Parameter(Mandatory=$true)] [string] $LibraryName
)
#Setup the context
$Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$Ctx.Credentials = $credentials
$srcWeb = $Ctx.Web
$childWebs = $srcWeb.Webs
$Ctx.Load($childWebs)
$Ctx.ExecuteQuery()
foreach ($childweb in $childWebs)
{
try
{
#Get the Library and Its Root Folder
$Library=$childweb.Lists.GetByTitle($LibraryName)
$Ctx.Load($Library)
$Ctx.Load($Library.RootFolder)
$Ctx.ExecuteQuery()
#Call the function to get Files of the Root Folder
if($childweb.Url.ToLower() -notlike "*ehcontactus*" -and $childweb.Url.ToLower() -notlike "*ehfaqapp*" -and $childweb.Url.ToLower() -notlike "*ehquicksearch*" -and $childweb.Url.ToLower() -notlike "*ehsiteapps*" -and $childweb.Url.ToLower() -notlike "*ehsitelist*" -and $childweb.Url.ToLower() -notlike "*ehwelcomeapp*" -and $childweb.Url.ToLower() -notlike "*ehimageviewer*")
{
Get-FilesFromFolder -Folder $Library.RootFolder -SubWeb $childweb.Url -MTitle $childweb.Title
}
}
catch{
write-host "Skipping the matterpsace as the library does not exists" -ForegroundColor Blue
}
}
}
#Config Parameters
#$SiteURL= "https://impigerspuat.sharepoint.com/sites/ELeave/Eleave1/adminuat#impigerspuat.onmicrosoft.com"
$LibraryName="Documents"
#$securePassword = Read-Host -Prompt "Enter your password: " -AsSecureString
#Call the function to Get All Files from a document library
if (-not ([string]::IsNullOrEmpty($fromDate)))
{
$startDate = Get-Date (Get-Date -Date $fromDate -Format 'dd/MM/yyyy')
}
else
{
$startDate = $null;
}
if (-not ([string]::IsNullOrEmpty($toDate)))
{
$endDate = Get-Date (Get-Date -Date $toDate -Format 'dd/MM/yyyy')
}
else
{
$endDate = $null
}
Get-SPODocLibraryFiles -SiteURL $srcUrl -LibraryName $LibraryName
Have you tried running just that function and giving it the parameters it's requesting in the function?
Copy the code into a WriteLog.ps1 file and then call the script file with the parameters.
ie.
Writelog.ps1 $srcUrl $username $fromDate $toDate $folderPath $csvPath
Obviously, inputting data in place of the variables.
FWIW, pulling relevant pieces of code out of someone else's scripts is a great skill to practice. Everything you want to do has been done before, but you might have to break down someone else's work before it fits your exact enviornment.
Unfortunately it looks like you have to do this the old fashion way. The problem is the author is outputting to the log (csv) as the files are being downloaded. As opposed to downloading to a staging area first...
I suggest setting an early break-point in the code then stepping through to see exactly how it's flowing. That should give you a general idea, and enough info to start writing refactored code.
Reverse engineering is always tough, be prepared it will be methodical exercise so say the least.
Bad news: this will be an iterative process, not a single 'solve'. Nothing "wrong" with that code, but there are a few design choices that make this a challenge. It's not indented consistently and it weaves through all the variable assignments in slightly different ways. Looks better than most of my code, I'm just telling you what makes it a challenge.
Good news: At least that WriteLog function is separate. And it's really just adding content to the .csv file defined in this variable assigned here:
$global:OutFilePath = -join ($csvPath,"\Documents.csv")
(line 20 in my copy)
*
RECOMMENDATION: (this is an approach, just a guide to your final solution)
Take that existing code and drop it in an IDE to help you visually. The Windows Powershell ISE is adequate, but I would highly recommend VSCode.
Comment out that last line:
Get-SPODocLibraryFiles -SiteURL $srcUrl -LibraryName $LibraryName
So you can retain any of the other context from the script you actually want to keep.
Create a separate function named something like:
function Get-FilesFromLocalFolder ($localdir, $SubWeb, $MTitle)
to use instead of the existing function Get-FilesFromFolder. That way you can iterate through whatever directories you need, get the files, and assign variables to pass as parameters. Then when you call WriteLog, it will look very similar. Those last two parameters ($SubWeb, $MTitle) are passed just because WriteLog needs them. You could make them your own labels, or you could remove them and make them optional in WriteLog.
You could start by hard-coding values in each of required parameters for the function, and then run it to see if the output is working.
It will take you some iterations (agree with #Steven) and it is definitely a valuable exercise (agree with #TheIdesOfMark). :)
I have the below function running in a logon script, which checks whether the user has the current version of IT Self Help.exe. If the current version is not present, then it should be copied onto the desktop from the $appsource folder:
function UrgentSupportApp {
$ErrorActionPreference = "Stop"
trap {Log-Error $_ $MyInvocation.MyCommand; Return}
$desktop = $env:USERPROFILE + '\Desktop\'
$apptarget = $desktop + 'IT Self Help.exe'
$appsource = '\\file\Administration\Unused\Apps\IT Support App\IT Self Help.exe'
# Remove the old version of the app "IT Help Request.exe"
$oldapps = Get-ChildItem $desktop -Filter *"Help Request.exe"
if ($oldapps.count -gt 0) {Remove-Item $oldapps.PSPath -Force}
# Copy the new version over if it is not already present
$currentversion = (Get-Command $appsource).FileVersionInfo.FileVersion
if (Test-Path $apptarget) {
if ((Get-Command $apptarget).FileVersionInfo.FileVersion -ne $currentversion) {
Copy-Item $appsource $desktop -Force ##### Line 981 #####
}
} else {
Copy-Item $appsource $desktop -Force
}
}
function Log-Error {
param (
$error,
[string]$sub,
$detail
)
$ErrorActionPreference = "Stop"
trap {Log-Error $_ $MyInvocation.MyCommand; Return}
$filename = "\\file\administration\Unused\LogonScriptErrors\$username - $sub - $computername - $(Get-Date -Format ddMMyyyy-HHmmss).log"
New-Item $filename -ItemType File -Value "Message: `r`n`t $($error.Exception.Message) `r`n `r`nPosition: `r`n`t $($error.InvocationInfo.PositionMessage) `r`n `r`nSub: `r`n`t $sub `r`n `r`nDetail: `r`n`t $detail"
}
For a couple of users, I am seeing this error come through on line 981, char 22 (see the comment above):
Could not find file 'C:\Users\USER.NAME\Desktop\IT Self Help.exe'.
At \\DC\NETLOGON\mainlogon.ps1:981 char:22
+ Copy-Item <<<< $appsource $desktop -Force
However
The file clearly can be found, as it made it through the fisrt If condition If (Test-Path $apptarget).
If the file couldn't be found, why would the script complain on that line, where we are not even looking for it?
What is this error trying to tell me? If the file could not be found, surely the script would just continue into the Else statement
I have the following Powershell code:
function readConfigData
{
$workingDir = (Get-Location).Path
$file = ""
if ($Global:USE_LOCAL_SERVER)
{
$file = $workingDir + '\Configs\Localhost.ini'
}
else
{
$file = $workingDir + '\Configs\' + $env:COMPUTERNAME + '.ini'
}
Write-Host 'INIFILE: ' $file
if (!$file -or ($file = ""))
{
throw [System.Exception] "Ini fil är inte satt."
}
if (!(Test-Path -Path $file))
{
throw [System.Exception] "Kan inte hitta ini fil."
}
}
readConfigData
How should I declare the local variable $file that can be passed to the function Test-Path.
My local variable $file get populated but then when I place it as argument to other function it's like it is out of scope.
I read the about scopes article but wasn't able to figure it out.
Currently I get the error:
INIFILE: D:\Projects\scripts\Configs\HBOX.ini Test-Path : Cannot bind argument to
parameter 'Path' because it is an empty string. At
D:\Projects\freelancer.com\nero2000\cmd script to
powershell\script.ps1:141 char:27
+ if (!(Test-Path -Path $file))
+ ~~~~~
+ CategoryInfo : InvalidData: (:) [Test-Path], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,Microsoft.PowerShell.Commands.TestPathCommand
if (!$file -or ($file = ""))
should be replaced by
if (!$file -or ($file -eq ""))
You assign $file to an empty string in the first if clause and therefore your variable is empty in the Test-Path call.
Edit: Also there are some alternatives: How can I check if a string is null or empty in PowerShell?
you could either use
if([string]::IsNullOrEmpty($file))
or even just
if(!$file)
As others have mentioned, you are unintentionally assigning a blank string to $file in your first if (!$file ... statement. That is really the root of your problem.
However, instead of:
if (!$file -or ($file = ""))
You could use this forumula, which I find explains itself better:
if([String]::IsNullOrEmpty($file))
I would define a function Get-ConfigFile to retrieve the config and add a switch for local server:
function Get-ConfigFile
{
Param(
[switch]$UseLocalServer
)
$workingDir = (Get-Location).Path
if ($UseLocalServer.IsPresent)
{
Join-Path $workingDir '\Configs\Localhost.ini'
}
else
{
Join-Path $workingDir ('\Configs\{0}.ini' -f $env:COMPUTERNAME)
}
}
I would also use the Join-Path cmdlet to join a path instead of string concatenations.
Now you can retrive the config file path using:
$configFile = Get-ConfigFile -UseLocalServer:$Global:USE_LOCAL_SERVER
And if needed, ensure that the file exists:
if (-not(Test-Path -Path $configFile))
{
throw [System.Exception] "Kan inte hitta ini fil."
}
Note:
Get-Location will give you the current powershell path (working location), if you want to get the path where your script is located, use this instead:
$workingDir = split-path -parent $MyInvocation.MyCommand.Definitio
I recently created a script that's able to create a chosen number of Hyper-V VMs from a "template", by exporting one VM and import it with different ids and names.
Everything is working smoothly, but the export takes quite some time. Is there any way to get an progress output from the export-vm? If I look in the hyper-v command center, I see that my VM is exported after i started the script, and it has a progress value, too, so somehow there seems to be a way to get the current progress...
It would also be nice to have a progress output for Import-VM, too, but that's not really important.
If you are interested, here is my current script. I know that the hard-coded paths and other ascpects aren't really nice, so please don't tell me anything about code style or something like this. First comes a working script, then comes nice code.
param(
[int]$Anzahl = 0,
[string]$BasisVM = 'Schulung',
[string]$ExportDir = 'C:\VMConf\Export\',
[string]$ExportConf = 'Schulung\Virtual Machines\0D444AF2-3E63-4ACF-867E-34440AA99C42.xml',
[string]$VMDir = 'C:\VMs\',
[string]$VMNamePrefix = 'Schulung'
)
if ($Anzahl -eq 0) {
$Anzahl = Read-Host "Bitte Anzahl der benötigten VMs eingeben"
}
"`nStarte Export der Vorlage..."
if (-Not(Test-Path $ExportDir)) {
New-Item -ItemType directory -Path $ExportDir | out-null
}
Export-VM -Name $BasisVM -Path $ExportDir
"Abgeschlossen.`n"
if (Test-Path $VMDir) {
$err = $null
$i = 1
while (-not($err)) {
if ($i -gt 1) {
"Alte VM Nummer $($i - 1) wurde gelöscht."
}
Remove-VM -Name $($VMNamePrefix + ("{0:D2}" -f $i)) -Force -ErrorVariable err -ErrorAction SilentlyContinue
$i++
}
Remove-Item -r $VMDir | out-null
New-Item -ItemType directory -Path $VMDir | out-null
}
"`n$Anzahl VMs werden erstellt..."
for ($i=1; $i -le $Anzahl; $i++) {
"`tErstelle VM Nummer $i..."
$name = ($VMNamePrefix + ("{0:D2}" -f $i))
$path = ($VMDir + $name)
$VM = Import-VM -Path $($ExportDir + $ExportConf) -VhdDestinationPath $($path + "/Virtual Hard Discs") -VirtualMachinePath $path -Copy -GenerateNewId
Rename-VM -VM $VM -NewName $name
"`t$VM Nummer $i wurde erzeugt."
}
"Abgeschlossen.`n"
"Aufräumen..."
if (Test-Path $ExportDir) {
Remove-Item -r $ExportDir | out-null
}
"Abgeschlossen."
You could start the export (and even imports) as a background job and then log the progress of the job. Sample code:
$ExportJob = Export-VM -Name $BasisVM -Path $ExportDir -AsJob;
while( $ExportJob.State -eq "Running" -or $ExportJob.State -eq "NotStarted")
{
Write-Output ("[Export] " + $($ExportJob.Progress.PercentComplete) + "% complete");
sleep(5);
}
if($ExportJob.State -ne "Completed")
{
Write-Error ("Export Job did not complete: " +$ExportJob.State);
throw $ExportJob.Error;
}
Is there any way to copy a really large file (from one server to another) in PowerShell AND display its progress?
There are solutions out there to use Write-Progress in conjunction with looping to copy many files and display progress. However I can't seem to find anything that would show progress of a single file.
Any thoughts?
It seems like a much better solution to just use BitsTransfer, it seems to come OOTB on most Windows machines with PowerShell 2.0 or greater.
Import-Module BitsTransfer
Start-BitsTransfer -Source $Source -Destination $Destination -Description "Backup" -DisplayName "Backup"
I haven't heard about progress with Copy-Item. If you don't want to use any external tool, you can experiment with streams. The size of buffer varies, you may try different values (from 2kb to 64kb).
function Copy-File {
param( [string]$from, [string]$to)
$ffile = [io.file]::OpenRead($from)
$tofile = [io.file]::OpenWrite($to)
Write-Progress -Activity "Copying file" -status "$from -> $to" -PercentComplete 0
try {
[byte[]]$buff = new-object byte[] 4096
[long]$total = [int]$count = 0
do {
$count = $ffile.Read($buff, 0, $buff.Length)
$tofile.Write($buff, 0, $count)
$total += $count
if ($total % 1mb -eq 0) {
Write-Progress -Activity "Copying file" -status "$from -> $to" `
-PercentComplete ([long]($total * 100 / $ffile.Length))
}
} while ($count -gt 0)
}
finally {
$ffile.Dispose()
$tofile.Dispose()
Write-Progress -Activity "Copying file" -Status "Ready" -Completed
}
}
Alternativly this option uses the native windows progress bar...
$FOF_CREATEPROGRESSDLG = "&H0&"
$objShell = New-Object -ComObject "Shell.Application"
$objFolder = $objShell.NameSpace($DestLocation)
$objFolder.CopyHere($srcFile, $FOF_CREATEPROGRESSDLG)
cmd /c copy /z src dest
not pure PowerShell, but executable in PowerShell and it displays progress in percents
I amended the code from stej (which was great, just what i needed!) to use larger buffer, [long] for larger files and used System.Diagnostics.Stopwatch class to track elapsed time and estimate time remaining.
Also added reporting of transfer rate during transfer and outputting overall elapsed time and overall transfer rate.
Using 4MB (4096*1024 bytes) buffer to get better than Win7 native throughput copying from NAS to USB stick on laptop over wifi.
On To-Do list:
add error handling (catch)
handle get-childitem file list as input
nested progress bars when copying multiple files (file x of y, % if
total data copied etc)
input parameter for buffer size
Feel free to use/improve :-)
function Copy-File {
param( [string]$from, [string]$to)
$ffile = [io.file]::OpenRead($from)
$tofile = [io.file]::OpenWrite($to)
Write-Progress `
-Activity "Copying file" `
-status ($from.Split("\")|select -last 1) `
-PercentComplete 0
try {
$sw = [System.Diagnostics.Stopwatch]::StartNew();
[byte[]]$buff = new-object byte[] (4096*1024)
[long]$total = [long]$count = 0
do {
$count = $ffile.Read($buff, 0, $buff.Length)
$tofile.Write($buff, 0, $count)
$total += $count
[int]$pctcomp = ([int]($total/$ffile.Length* 100));
[int]$secselapsed = [int]($sw.elapsedmilliseconds.ToString())/1000;
if ( $secselapsed -ne 0 ) {
[single]$xferrate = (($total/$secselapsed)/1mb);
} else {
[single]$xferrate = 0.0
}
if ($total % 1mb -eq 0) {
if($pctcomp -gt 0)`
{[int]$secsleft = ((($secselapsed/$pctcomp)* 100)-$secselapsed);
} else {
[int]$secsleft = 0};
Write-Progress `
-Activity ($pctcomp.ToString() + "% Copying file # " + "{0:n2}" -f $xferrate + " MB/s")`
-status ($from.Split("\")|select -last 1) `
-PercentComplete $pctcomp `
-SecondsRemaining $secsleft;
}
} while ($count -gt 0)
$sw.Stop();
$sw.Reset();
}
finally {
write-host (($from.Split("\")|select -last 1) + `
" copied in " + $secselapsed + " seconds at " + `
"{0:n2}" -f [int](($ffile.length/$secselapsed)/1mb) + " MB/s.");
$ffile.Close();
$tofile.Close();
}
}
Not that I'm aware of. I wouldn't recommend using copy-item for this anyway. I don't think it has been designed to be robust like robocopy.exe to support retry which you would want for extremely large file copies over the network.
i found none of the examples above met my needs, i wanted to copy a directory with sub directories, the problem is my source directory had too many files so i quickly hit the BITS file limit (i had > 1500 file) also the total directory size was quite large.
i found a function using robocopy that was a good starting point at https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress/, however i found it wasn't quite robust enough, it didn't handle trailing slashes, spaces gracefully and did not stop the copy when the script was halted.
Here is my refined version:
function Copy-ItemWithProgress
{
<#
.SYNOPSIS
RoboCopy with PowerShell progress.
.DESCRIPTION
Performs file copy with RoboCopy. Output from RoboCopy is captured,
parsed, and returned as Powershell native status and progress.
.PARAMETER Source
Directory to copy files from, this should not contain trailing slashes
.PARAMETER Destination
DIrectory to copy files to, this should not contain trailing slahes
.PARAMETER FilesToCopy
A wildcard expresion of which files to copy, defaults to *.*
.PARAMETER RobocopyArgs
List of arguments passed directly to Robocopy.
Must not conflict with defaults: /ndl /TEE /Bytes /NC /nfl /Log
.PARAMETER ProgressID
When specified (>=0) will use this identifier for the progress bar
.PARAMETER ParentProgressID
When specified (>= 0) will use this identifier as the parent ID for progress bars
so that they appear nested which allows for usage in more complex scripts.
.OUTPUTS
Returns an object with the status of final copy.
REMINDER: Any error level below 8 can be considered a success by RoboCopy.
.EXAMPLE
C:\PS> .\Copy-ItemWithProgress c:\Src d:\Dest
Copy the contents of the c:\Src directory to a directory d:\Dest
Without the /e or /mir switch, only files from the root of c:\src are copied.
.EXAMPLE
C:\PS> .\Copy-ItemWithProgress '"c:\Src Files"' d:\Dest /mir /xf *.log -Verbose
Copy the contents of the 'c:\Name with Space' directory to a directory d:\Dest
/mir and /XF parameters are passed to robocopy, and script is run verbose
.LINK
https://keithga.wordpress.com/2014/06/23/copy-itemwithprogress
.NOTES
By Keith S. Garner (KeithGa#KeithGa.com) - 6/23/2014
With inspiration by Trevor Sullivan #pcgeek86
Tweaked by Justin Marshall - 02/20/2020
#>
[CmdletBinding()]
param(
[Parameter(Mandatory=$true)]
[string]$Source,
[Parameter(Mandatory=$true)]
[string]$Destination,
[Parameter(Mandatory=$false)]
[string]$FilesToCopy="*.*",
[Parameter(Mandatory = $true,ValueFromRemainingArguments=$true)]
[string[]] $RobocopyArgs,
[int]$ParentProgressID=-1,
[int]$ProgressID=-1
)
#handle spaces and trailing slashes
$SourceDir = '"{0}"' -f ($Source -replace "\\+$","")
$TargetDir = '"{0}"' -f ($Destination -replace "\\+$","")
$ScanLog = [IO.Path]::GetTempFileName()
$RoboLog = [IO.Path]::GetTempFileName()
$ScanArgs = #($SourceDir,$TargetDir,$FilesToCopy) + $RobocopyArgs + "/ndl /TEE /bytes /Log:$ScanLog /nfl /L".Split(" ")
$RoboArgs = #($SourceDir,$TargetDir,$FilesToCopy) + $RobocopyArgs + "/ndl /TEE /bytes /Log:$RoboLog /NC".Split(" ")
# Launch Robocopy Processes
write-verbose ("Robocopy Scan:`n" + ($ScanArgs -join " "))
write-verbose ("Robocopy Full:`n" + ($RoboArgs -join " "))
$ScanRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $ScanArgs
try
{
$RoboRun = start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList $RoboArgs
try
{
# Parse Robocopy "Scan" pass
$ScanRun.WaitForExit()
$LogData = get-content $ScanLog
if ($ScanRun.ExitCode -ge 8)
{
$LogData|out-string|Write-Error
throw "Robocopy $($ScanRun.ExitCode)"
}
$FileSize = [regex]::Match($LogData[-4],".+:\s+(\d+)\s+(\d+)").Groups[2].Value
write-verbose ("Robocopy Bytes: $FileSize `n" +($LogData -join "`n"))
#determine progress parameters
$ProgressParms=#{}
if ($ParentProgressID -ge 0) {
$ProgressParms['ParentID']=$ParentProgressID
}
if ($ProgressID -ge 0) {
$ProgressParms['ID']=$ProgressID
} else {
$ProgressParms['ID']=$RoboRun.Id
}
# Monitor Full RoboCopy
while (!$RoboRun.HasExited)
{
$LogData = get-content $RoboLog
$Files = $LogData -match "^\s*(\d+)\s+(\S+)"
if ($null -ne $Files )
{
$copied = ($Files[0..($Files.Length-2)] | ForEach-Object {$_.Split("`t")[-2]} | Measure-Object -sum).Sum
if ($LogData[-1] -match "(100|\d?\d\.\d)\%")
{
write-progress Copy -ParentID $ProgressParms['ID'] -percentComplete $LogData[-1].Trim("% `t") $LogData[-1]
$Copied += $Files[-1].Split("`t")[-2] /100 * ($LogData[-1].Trim("% `t"))
}
else
{
write-progress Copy -ParentID $ProgressParms['ID'] -Complete
}
write-progress ROBOCOPY -PercentComplete ($Copied/$FileSize*100) $Files[-1].Split("`t")[-1] #ProgressParms
}
}
} finally {
if (!$RoboRun.HasExited) {Write-Warning "Terminating copy process with ID $($RoboRun.Id)..."; $RoboRun.Kill() ; }
$RoboRun.WaitForExit()
# Parse full RoboCopy pass results, and cleanup
(get-content $RoboLog)[-11..-2] | out-string | Write-Verbose
remove-item $RoboLog
write-output ([PSCustomObject]#{ ExitCode = $RoboRun.ExitCode })
}
} finally {
if (!$ScanRun.HasExited) {Write-Warning "Terminating scan process with ID $($ScanRun.Id)..."; $ScanRun.Kill() }
$ScanRun.WaitForExit()
remove-item $ScanLog
}
}
Hate to be the one to bump an old subject, but I found this post extremely useful. After running performance tests on the snippets by stej and it's refinement by Graham Gold, plus the BITS suggestion by Nacht, I have decuded that:
I really liked Graham's command with time estimations and speed readings.
I also really liked the significant speed increase of using BITS as my transfer method.
Faced with the decision between the two... I found that Start-BitsTransfer supported Asynchronous mode. So here is the result of my merging the two.
function Copy-File {
# ref: https://stackoverflow.com/a/55527732/3626361
param([string]$From, [string]$To)
try {
$job = Start-BitsTransfer -Source $From -Destination $To `
-Description "Moving: $From => $To" `
-DisplayName "Backup" -Asynchronous
# Start stopwatch
$sw = [System.Diagnostics.Stopwatch]::StartNew()
Write-Progress -Activity "Connecting..."
while ($job.JobState.ToString() -ne "Transferred") {
switch ($job.JobState.ToString()) {
"Connecting" {
break
}
"Transferring" {
$pctcomp = ($job.BytesTransferred / $job.BytesTotal) * 100
$elapsed = ($sw.elapsedmilliseconds.ToString()) / 1000
if ($elapsed -eq 0) {
$xferrate = 0.0
}
else {
$xferrate = (($job.BytesTransferred / $elapsed) / 1mb);
}
if ($job.BytesTransferred % 1mb -eq 0) {
if ($pctcomp -gt 0) {
$secsleft = ((($elapsed / $pctcomp) * 100) - $elapsed)
}
else {
$secsleft = 0
}
Write-Progress -Activity ("Copying file '" + ($From.Split("\") | Select-Object -last 1) + "' # " + "{0:n2}" -f $xferrate + "MB/s") `
-PercentComplete $pctcomp `
-SecondsRemaining $secsleft
}
break
}
"Transferred" {
break
}
Default {
throw $job.JobState.ToString() + " unexpected BITS state."
}
}
}
$sw.Stop()
$sw.Reset()
}
finally {
Complete-BitsTransfer -BitsJob $job
Write-Progress -Activity "Completed" -Completed
}
}
This recursive function copies files and directories recursively from source path to destination path
If file already exists on destination path, it copies them only with newer files.
Function Copy-FilesBitsTransfer(
[Parameter(Mandatory=$true)][String]$sourcePath,
[Parameter(Mandatory=$true)][String]$destinationPath,
[Parameter(Mandatory=$false)][bool]$createRootDirectory = $true)
{
$item = Get-Item $sourcePath
$itemName = Split-Path $sourcePath -leaf
if (!$item.PSIsContainer){ #Item Is a file
$clientFileTime = Get-Item $sourcePath | select LastWriteTime -ExpandProperty LastWriteTime
if (!(Test-Path -Path $destinationPath\$itemName)){
Start-BitsTransfer -Source $sourcePath -Destination $destinationPath -Description "$sourcePath >> $destinationPath" -DisplayName "Copy Template file" -Confirm:$false
if (!$?){
return $false
}
}
else{
$serverFileTime = Get-Item $destinationPath\$itemName | select LastWriteTime -ExpandProperty LastWriteTime
if ($serverFileTime -lt $clientFileTime)
{
Start-BitsTransfer -Source $sourcePath -Destination $destinationPath -Description "$sourcePath >> $destinationPath" -DisplayName "Copy Template file" -Confirm:$false
if (!$?){
return $false
}
}
}
}
else{ #Item Is a directory
if ($createRootDirectory){
$destinationPath = "$destinationPath\$itemName"
if (!(Test-Path -Path $destinationPath -PathType Container)){
if (Test-Path -Path $destinationPath -PathType Leaf){ #In case item is a file, delete it.
Remove-Item -Path $destinationPath
}
New-Item -ItemType Directory $destinationPath | Out-Null
if (!$?){
return $false
}
}
}
Foreach ($fileOrDirectory in (Get-Item -Path "$sourcePath\*"))
{
$status = Copy-FilesBitsTransfer $fileOrDirectory $destinationPath $true
if (!$status){
return $false
}
}
}
return $true
}
Sean Kearney from the Hey, Scripting Guy! Blog has a solution I found works pretty nicely.
Function Copy-WithProgress
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
$Source,
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true,
Position=0)]
$Destination
)
$Source=$Source.tolower()
$Filelist=Get-Childitem "$Source" –Recurse
$Total=$Filelist.count
$Position=0
foreach ($File in $Filelist)
{
$Filename=$File.Fullname.tolower().replace($Source,'')
$DestinationFile=($Destination+$Filename)
Write-Progress -Activity "Copying data from '$source' to '$Destination'" -Status "Copying File $Filename" -PercentComplete (($Position/$total)*100)
Copy-Item $File.FullName -Destination $DestinationFile
$Position++
}
}
Then to use it:
Copy-WithProgress -Source $src -Destination $dest
Trevor Sullivan has a write-up on how to add a command called Copy-ItemWithProgress to PowerShell on Robocopy.