I have script like below:
foreach ($var1 in (gc 1.txt)){
// my code logic
}
Here 1.txt file contains list of values like abc, xyz, pqr etc..,
If due to any of the script issues/ctrl+c stopped the script, need to restart the script from last stopped session.
To be clear if script has stopped at processing file at 'xyz' and when i restart the script it should process the logic from 'xyz' only but shouldn't restart from 'abc' again.
Please guide me to achieve this logic.
Thanks in advance,
Pavan kumar D
You need to add some counters and keep track of last iteration using and index file.
Then make sure that you start reading the file where you was interrupted using select -Skip
<#
.NOTE
The file index.txt is created at the first run and is used to store
the last line accessed of the file to process.
The index read from index.txt is used to skip by already processed
lines at the start of the foreach.
If the index doesn't need to be stored between session, then use
a $global.index instead to speed up the script, instead of an index
file (or a RAM-drive, not very common any more).
At each succefull iteration, the index is incremented and then stored in
index.txt.
When starting the stored index.txt is compared with the numbers of
lines in the file and will through an error if it's passed End Of File.
Make sure to **clear** the index.txt before starting the file process
fresh.
#>
#initialize counters
[int]$StartLine = Get-Content .\index.txt -ErrorAction SilentlyContinue
if (-not $StartLine){[int]$StartLine = 0} # First run will have no index file
$Index = $StartLine
[int]$LastLineOfFile = (get-content 1.txt).count - 1 # Arrays starts at 0
if ($Index -gt $LastLineOfFile){# Don't start if index passed EOF at last run
Write-Error "Index passed end of file"
return
}
foreach ($var in (Get-Content 1.txt | Select -Skip $StartLine)){
"New loop: $Index" | Out-Host # will start empty
"Processing value: $var" | Out-Host
####
# Processing here
####
"Done processing value: $var" | Out-Host
$Index++
$Index > index.txt
"Index incremented" | Out-Host
}
For a good article of using RAM drive, see, i.e. How to Create RAM Disk in Windows 10 for Super-Fast Read and Write Speeds
Windows Server do support RAM disks natively (kind of) by using the iSCSI Target Server.
See, How to Create a RAM Disk on Windows Server?
Related
I'm trying to fetch some registry parameters for about 4'000 machines with PowerShell. There's by no mean any chance that all machines are going to be up at the same time so I'd like to be able to save the list of machines that were already backed up for the script to only get the parameters from not already backed up machines.
I made a CSV file in the form of Machine,Status where the Machine column stores machines names and Status is supposedly either equal to 0 if the script hasn't run yet on this machine and 1 if the machine has already been backed up.
I'm already successfully parsing the machines names from the CSV where Status = 0 and running my backup script for all machines that are up at script run time but I can't figure out how to set the variable Status to 1 back to the CSV (or from what I understood after some reading, to a temporary CSV that will later replace the original one).
My code is like:
$csv = Import-Csv -Path ./list.csv
foreach ($line in $csv) {
$status = $line.Status
$machine = $line.Machine
if ($status -eq "0") {
[REG BACKUP SCRIPT]
$status = 1 #Set the machine as done, but how to put it back to the CSV ?
Export-Csv -Path ./list-new.csv
I don't know where to put Export-Csv and after some attempts
Thanks
I've inherited a deployment script that reads through files with specific names and extensions deployed to our on-prem web servers, looks line-by-line in those text files for string matches that signify they need replacing, and then inserts, in this case, the correct database and catalog names for the specific server being deployed to.
Unfortunately, though the script works fine On My Computer (c)1984, it's not working when run by TFS as part of the deploy process.
I've added an embarrassing amount of debug info, but haven't been able to track down the issue.
Code
Here's the pertinent code:1
$getFilesParams = #{
Path = $SearchDirectory
Include = #(
"*.asa", "*.asp",
"*.aspx", "*.config",
"*.inc", "*.ini",
"appsettings.json"
)
Exclude = "Web.*.config"
Recurse = $True
}
$files = #(Get-ChildItem #getFilesParams)
foreach ($f in $files) {
# DEBUG STARTS HERE
Write-Feedback "`tFile to search for Data Source: $f" -LogFile "$LogFile"
if ($f.Name -eq "appsettings.json") {
try {
Write-Feedback "`t`tFound the appsettings.json file: $($f.Length) $($f.LastWriteTime) $($f.Name)" -LogFile $LogFile
Get-Content $f | % {
Write-Feedback "!!!! Line: $_"
}
Select-String $f -Pattern "Data Source" | % { Write-Feedback "`t`t`tFound data source: $_" }
# was suspicious it'd work the first but not the second time. No, it fails each time I Get-Content.
Get-Content $f | % {
Write-Feedback "#### Line: $_"
}
}
catch {
Write-Feedback "An error occurred with appsettings.json:" -LogFile $LogFile
Write-Feedback $_ -LogFile $LogFile
}
}
# DEBUG ENDS
}
$files = $files |
Select-String -Pattern "Data Source" |
Group-Object path |
Select-Object name
$count = $files.count
if ($count)
{
#"
Found $count files....
Processing:
"# | Write-Feedback -LogFile $LogFile
# etc etc he fixes the cable
}
else
{
' Did not find any databases catalogs!' | Write-Feedback -LogFile $LogFile
}
Then we go line by line through the files in $files. The problem is that my appsettings.json file, which does contain Data Source (okay, it's lowercase right now -- data source), doesn't get captured and no replacements are made.
Note: Write-Feedback is a convenience function that writes to the console (Write-Host) and to a file, if one is given.
Local output
When I run locally, I get what I'm expecting (edited a bit for brevity):
File to search for Data Source: C:\path\appsettings.json
Found the appsettings.json file: 993 01/12/2022 13:04:52 appsettings.json
!!!! Line: "SomeJsonKey": "data source=localhost;initial catalog=SomeDb;integrated security=True",
Found datasource: C:\path\appsettings.json:9: "SomeJsonKey": "data source=localhost;initial catalog=SomeDb;integrated security=True",
#### Line: "SomeJsonKey": "data source=localhost;initial catalog=SomeDb;integrated security=True",
Found 1 files....
Processing:
C:\path\appsettings.json....
Production output
But when it's run as part of the deployment, I get...
File to search for Data Source: E:\path\appsettings.json
Found the appsettings.json file: 762 01/14/2022 15:15:02 appsettings.json
Did not find any databases catalogs!
So it sees appsettings.json , it even knows appsettings.json has a length (they are different files, so the two lengths here aren't an issue), but it won't Get-Content, much less find the line with Data Source in it.
Notepad++ says the file is ANSI, which is fine, I think. There're no extended characters, so that's the same as UTF8, which is what I expected. Don't think that'd break Get-Content.
To be clear, Not necessarily looking for an answer. Just wondering what my next debug(s) step would be.
I've looked up a little about permissions, though I'm not sure how I'd tell which entry from (Get-Acl $Outfile).Access represents the process that's currently running. But I also would expect an exception of some kind if it can't read the file.
Just for fun, here's a picture of the release UI for the TFS server this is running on. If you've seen it before, you'll get it.
I was hoping I could figure this out through straight PowerShell, but
I can check things out there to a degree if that's useful. That said, I don't have full perms on the TFS site. I probably have more running as... whoever I'm running as when the PowerShell (third step) is executed.
(sorry for all the redaction; almost certainly not necessary, but better safe than sorry (sorrier?), etc?)
1 I realize this code can be optimized. Please ignore that; it's, again, inherited, and I'm trying to get it to work with a minimum of churn.
I have a power shell script that keeps running and I already export the output of this script into a notepad file and I always append it to keep updating itself.
Now, I want the output to 2 plain-text files.
1 which would have my appended output, and 2 which has only the latest data.
How can I export the same output to 2 text files at same time?
Below is a snippet of my script.
$(
write-host 'Checking Status.........'
GC .\file.txt | %{.\Status.ps1 $_}
) *>&1 > WINRMStatus.txt -Append
You can combine an appending redirection (>>) with Tee-Object:
& {
write-host 'Checking Status.........'
GC .\file.txt | %{.\Status.ps1 $_}
} *>&1 | Tee-Object OtherFile.txt >> WINRMStatus.txt
Also note that I've wrapped your statements in a script block ({ ... }) invoked with the call operator (&), which - unlike $(...), the subexpression operator - preserves the streaming behavior of the enclosed commands (sends their output to the pipeline one by one instead of collecting everything in memory first).
Summary
I am currently tasked with migrating around 6TB of data to a cloud server, and am trying to optimise how fast this can be done.
I would use standard Robocopy to do this usually, but there is a requirement that I am to only transfer files that are present in a filetable in SQL, and not the entire directories (due to a lot of junk being inside these folders that we do not want to migrate).
What I have tried
Feeding in individual files from an array into Robocopy is unfeasibly slow, as Robocopy instances were being started sequentially for each file, so I tried to speed up this process in 2 ways.
It was pointless to have /MT set above 1 if only one file was being transferred, so I attempted to simulate the multithreading feature. I did this by utilising the new ForEach-Object –Parallel feature in PowerShell 7.0, and setting the throttle limit to 4. With this, I was able to pass the array in and run 4 Robocopy jobs in parallel (still starting and stopping for each file), which increased speed a bit.
Secondly, I split the array into 4 equal arrays, and ran the above function across each array as a job, which again increased the speed by quite a bit. For clarity, I had equal 4 arrays fed to 4 ForEach-Object -Parallel code blocks that were running 4 Robocopy instances, so a total of 16 Robocopy instances at once.
Issues
I encountered a few problems.
My simulation of the multithreading feature did not behave in the way that the /MT flag works in Robocopy. When examining the processes running, my code executes 16 instances of Robocopy at once, whereas the normal /MT:16 flag of Robocopy would only kick off one Robocopy instance (but still be multithreading).
Secondly, the code causes a memory leak. The memory usage starts to increase when the jobs and accumulates over time, until a large portion of memory is being utilised. When the jobs complete, the memory usage is still high until I close PowerShell and the memory is released. Normal Robocopy did not do this.
Finally, I decided to compare the time taken for my method, and then a standard Robocopy of the entire testing directory, and the normal Robocopy was still over 10x faster, and had a better success rate (a lot of the files weren’t copied over with my code, and a lot of the time I was receiving error messages that the files were currently in use and couldn’t be Robocopied, presumably because they were in the process of being Robocopied).
Are there any faster alternatives, or is there a way to manually create a multithreading instance of robocopy that would perform like the /MT flag of the standard robocopy? I appreciate any insight/alternative ways of looking at this. Thanks!
#Item(0) is the Source excluding the filename, Item(2) is the Destination, Item(1) is the filename
$robocopy0 = $tables.Tables[0].Rows
$robocopy1 = $tables.Tables[1].Rows
$robocopy0 | ForEach-Object -Parallel {robocopy $_.Item(0) $_.Item(2) $_.Item(1) /e /w:1 /r:1 /tee /NP /xo /mt:1 /njh /njs /ns
} -ThrottleLimit 4 -AsJob
$robocopy1 | ForEach-Object -Parallel {robocopy $_.Item(0) $_.Item(2) $_.Item(1) /e /w:1 /r:1 /tee /NP /xo /mt:1 /njh /njs /ns
} -ThrottleLimit 4 -AsJob
#*8 for 8 arrays
RunspaceFactory multithreading might be optimally suited for this type of work--with one HUGE caveat. There are quite a few articles out on the net about it. Essentially you create a scriptblock that takes parameters for the source file to copy and the destination to write to and uses those parameters to execute robocopy against it. You create individual PowerShell instances to execute each variant of the scriptblock and append it to the RunspaceFactory. The RunspaceFactory will queue up the jobs and work against the probably millions of jobs X number at a time, where X is equal to the number of threads you allocate for the factory.
CAVEAT: First and foremost, to queue up millions of jobs relative to the probable millions of files you have across 6TB, you'll likely need monumental amounts of memory. Assuming an average path length for source and destination of 40 characters (probably very generous) * a WAG of 50 million files is nearly 4GB in memory by itself, which doesn't include object structural overhead, the PowerShell instances, etc. You can overcome this either breaking up the job into smaller chunks or use a server with 128GB RAM or better. Additionally, if you don't terminate the jobs once they've been processed, you'll also experience what appears to be a memory leak but is just your jobs producing information that you're not closing when completed.
Here's a sample from a recent project I did migrating files from an old domain NAS to a new domain NAS -- I'm using Quest SecureCopy instead of RoboCopy but you should be able to easily replace those bits:
## MaxThreads is an arbitrary number I use relative to the hardware I have available to run jobs I'm working on.
$FileRSpace_MaxThreads = 15
$FileRSpace = [runspacefactory]::CreateRunspacePool(1, $FileRSpace_MaxThreads, ([System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()), $Host)
$FileRSpace.ApartmentState = 'MTA'
$FileRSpace.Open()
## The scriptblock that does the actual work.
$sb = {
param(
$sp,
$dp
)
## This is my output object I'll emit through STDOUT so I can consume the status of the job in the main thread after each instance is completed.
$n = [pscustomobject]#{
'source' = $sp
'dest' = $dp
'status' = $null
'sdtm' = [datetime]::Now
'edtm' = $null
'elapsed' = $null
}
## Remove the Import-Module and SecureCopy cmdlet and replace it with the RoboCopy version
try {
Import-Module "C:\Program Files\Quest\Secure Copy 7\SCYPowerShellCore.dll" -ErrorAction Stop
Start-SecureCopyJob -Database "C:\Program Files\Quest\Secure Copy 7\SecureCopy.ssd" -JobName "Default" -Source $sp -Target $dp -CopySubFolders $true -Quiet $true -ErrorAction Stop | Out-Null
$n.status = $true
} catch {
$n.status = $_
}
$n.edtm = [datetime]::Now
$n.elapsed = ("{0:N2} minutes" -f (($n.edtm - $n.sdtm).TotalMinutes))
$n
}
## The array to hold the individual runspaces and ulitimately iterate over to watch for completion.
$FileWorkers = #()
$js = [datetime]::now
log "Job starting at $js"
## $peers is a [pscustomobject] I precreate that just contains every source (property 's') and the destination (property 'd') -- modify to suit your needs as necessary
foreach ($c in $peers) {
try {
log "Configuring migration job for '$($c.s)' and '$($c.d)'"
$runspace = [powershell]::Create()
[void]$runspace.AddScript($sb)
[void]$runspace.AddArgument($c.s)
[void]$runspace.AddArgument($c.d)
$runspace.RunspacePool = $FileRSpace
$FileWorkers += [pscustomobject]#{
'Pipe' = $runspace
'Async' = $runspace.BeginInvoke()
}
log "Successfully created a multi-threading job for '$($c.s)' and '$($c.d)'"
} catch {
log "An error occurred creating a multi-threading job for '$($c.s)' and '$($c.d)'"
}
}
while ($FileWorkers.Async.IsCompleted -contains $false) {
$Completed = $FileWorkers | ? { $_.Async.IsCompleted -eq $true }
[pscustomobject]#{
'Numbers' = ("{0}/{1}" -f $Completed.Count, $FileWorkers.Count)
'PercComplete' = ("{0:P2}" -f ($Completed.Count / $FileWorkers.Count))
'ElapsedMins' = ("{0:N2}" -f ([datetime]::Now - $js).TotalMinutes)
}
$Completed | % { $_.Pipe.EndInvoke($_.Async) } | Export-Csv -NoTypeInformation ".\$($DtmStamp)_SecureCopy_Results.csv"
Start-Sleep -Seconds 15
}
## This is to handle a race-condition where the final job(s) aren't completed before the sleep but do when the while is re-eval'd
$FileWorkers | % { $_.Pipe.EndInvoke($_.Async) } | Export-Csv -NoTypeInformation ".\$($DtmStamp)_SecureCopy_Results.csv"
Suggested strategies if you don't have a beefy server to queue up all the jobs simultaneously is to either batch out the files in statically sized blocks (e.g. 100,000 or whatever your hw can take) or you could group files together to send to each script block (e.g. 100 files per scriptblock) which would minimize the number of jobs to queue up in the runspace factory (but would require some code change).
HTH
Edit 1: To Address constructing the input object I'm using
$destRoot = '\\destinationserver.com\share'
$peers = #()
$children = #()
$children += (get-childitem '\\sourceserver\share' -Force) | Select -ExpandProperty FullName
foreach ($c in $children) {
$peers += [pscustomobject]#{
's' = $c
'd' = "$($destRoot)\$($c.Split('\')[3])\$($c | Split-Path -Leaf)"
}
}
In my case, I was taking stuff from \server1\share1\subfolder1 and moving it to something like \server2\share1\subfolder1\subfolder2. So in essence, all the '$peers' array is doing is constructing an object that took in the fullname of the source target and constructing the corresponding destination path (since the source/dest server names are different and possibly share name too).
You don't have to do this, you can dynamically construct the destination and just loop through the source folders. I perform this extra step because now I have a two property array that I can verify is pre-constructed accurately as well as perform tests to ensure things exist and are accessible.
There is a lot of extra bloat in my script due to custom objects meant to give me output from each thread put into the multi-threader so I can see the status of each copy attempt--to track things like folders that were successful or not, how long it took to perform that individual copy, etc. If you're using robocopy and dumping the results to a text file, you may not need this. If you want me to pair down script to it's barebone components just to get things multi-threading, I can do that if you like.
I know PowerShell is up to v5, but as I am new to PowerShell, I've been looking through Stack Overflow to generate the script I have. I've found that I need a generic non-version specific way of accomplishing this process...
Here is the issue - Step 1 - I'm pulling application installation locations information from the registry and am using a temporary file to house the results.
dir "HKLM:\SOFTWARE\Wow6432Node\companyname" | Get-ItemProperty | Select installdir | Out-File "$env:USERPROFILE\Desktop\KDI-Admin\Export\$env:COMPUTERNAME-SC-Installs.txt"
This provides me a list of installation directories for the company's software that is installed on a particular machine. I then want to take these results, append *.config to each line, as well as taking these results and appending *.xml to each line, and output to a new text file.
The input for the process would be the contents of the initial results file, and the output file should have each line listed in the first results, added to the final results file, once appended with *.xml, and once appended with *.config.
The net effect I am looking for is the creation of a #file for a 7z command. I am attempting this by using the following -
(Get-Content "$env:USERPROFILE\Desktop\KDI-Admin\Export\$env:COMPUTERNAME-SC-Installs.txt") -replace '\S+$','$&*.config' | Out-File "$env:USERPROFILE\Desktop\KDI-Admin\Export\$env:COMPUTERNAME-SC-config.txt" -Encoding utf8
(Get-Content "$env:USERPROFILE\Desktop\KDI-Admin\Export\$env:COMPUTERNAME-SC-Installs.txt") -replace '\S+$','$&*.xml' | Out-File "$env:USERPROFILE\Desktop\KDI-Admin\Export\$env:COMPUTERNAME-SC-config.txt" -Append -Encoding utf8
However, I'm only getting one line that has *.xml and one line that has *.config appended -
After getting this far, I'm thinking that some for-each loop is needed, but I'm not getting anywhere with what I have tried adapting from here. I'm looking now for some way to combine the three lines into one function, if that is possible, and eliminate the temporary file step in the first command, by reading and outputting in the same step. This would also need to remove the "installdir" and "----------" lines from the output. Anyone have some ideas and maybe examples?
Taken your above command dir "HKLM:\SOFTWARE\Wow6432Node\companyname" | Get-ItemProperty | Select installdir | Out-File "$env:USERPROFILE\Desktop\KDI-Admin\Export\$env:COMPUTERNAME-SC-Installs.txt" you could put the result of your query into a variable $result:
$result = dir "HKLM:\SOFTWARE\Wow6432Node\microsoft" | Get-ItemProperty | Select installdir;
From there you can easily loop through the array, skipping empty ones and process the rest of it:
foreach($path in $result.installdir)
{
# skip empty paths
if([string]::IsNullOrWhiteSpace($path)) { continue; }
# now do your processing ...
$path;
}
Is this what you were asking for?