Powershell: Some simple numbering - powershell

So I wanted to get a simple "this is where we're up to" numbering to show in the host when running a powershell script. I created the below to mess around with this:
$musicList = Get-ChildItem "C:\Users\rlyons\Music" -Recurse -File | Select-Object FullName
$outfile = #{}
$num = 1
foreach ($sound in $musicList) {
$file = $sound.FullName.ToString()
$md5 = Get-FileHash $file -Algorithm MD5
$outFile.$file += $md5.Hash
Write-Output "Processed file $num"
$num ++
}
This works great! Except it gives me:
Processed file 1
Processed file 2
Processed file 3
Processed file 4
Processed file 5
etc etc. What I wanted was for the screen to clear every time it processed a file, so all I wanted to see was the number of the file processed change at the end of the line written to the host.
I tried adding in the good old Clear-Host after the $num ++ and also at the start of the foreach loop, but all I get is a flickering line, and sometimes something partially ledgible. Yes, this is down to how fast the files are being processed, but I was wondering if there is a trick to this? To keep the Processed file bit continously on screen, but have the number increment upwards?
Even better would be if it didn't clear the screen, so you could see all previously written outputs but have this one line refresh? Is this possible?
Note: I'm trying this in VS Code, Powershell 5.1

Per the comments, Write-Progress is a much better fit for this because it gives you the output you want but without affecting the actual output stream of the script (e.g what is returned to any potential follow on command you might write).
Here's how you might want to use Write-Progress that will also show the percentage of completion:
$musicList = Get-ChildItem "C:\Users\rlyons\Music" -Recurse -File | Select-Object FullName
$outfile = #{}
$num = 1
foreach ($sound in $musicList) {
$file = $sound.FullName.ToString()
$md5 = Get-FileHash $file -Algorithm MD5
$outFile.$file += $md5.Hash
Write-Progress -Activity "Processed file $num of $($musiclist.count)" -PercentComplete ($num / $musiclist.count * 100)
$num ++
}

Related

Why is other powershell script run twice?

I have a script that checks whether disk has a certain amount of free space. If not, a pop-up appears asking for a yes or no. If yes, then an alarm is set to 1 and then another script that deletes files from folder runs. My issue is that it seems to delete twice the number specified in the script.
Main script:
$limit_low = 0.1 # låg gräns 10%
$DiskD = Get-PSDrive D | Select-Object Used,Free | Write-Output
$DiskD_use = [math]::Round(($DiskD.Free / ($DiskD.Used + $DiskD.Free)),2)
if( $DiskD_use -le $limit_low ) {
Write-Host "RDS-server har för lite utrymme på disk D $diskD_use < $limit_low" -ForegroundColor Red -BackgroundColor Yellow
$ButtonType = 4
$Timeout = 60
$Confirmation = New-Object -ComObject wscript.shell
$ConfirmationAnswer = $Confirmation.popup("Clear disk space?",$Timeout,"No space",$ButtonType)
If( $ConfirmationAnswer -eq 6 ) {
Write-Host "Kör script Diskspace.ps1 under P:\backupscripts"
& c:\dynamics\app\JDSend.exe "/UDP /LOG:c:\dynamics\app\Fixskick.log /TAG:Lunsc2:K_PROCESS_LARM_DISKUTRYMME "1""
& P:\BackupScripts\Delete_archives_test.ps1 # here i call the other script
} else {
Write-Host "Gör ingenting"
& c:\dynamics\app\JDSend.exe "/UDP /LOG:c:\dynamics\app\Fixskick.log /TAG:Lunsc2:K_PROCESS_LARM_DISKUTRYMME "0""
}
}
Other script:
# List all txt-files in directory, sort them och select the first 10, then delete
Get-ChildItem -Path c:\temp -File Archive*.txt | Sort-Object | Select-Object -First 10 | Remove-Item -Force
Cheers
EDIT
So it would be enough to enclose the first statement like this:
(Get-ChildItem -Path c:\temp -File Archive*.txt) | Sort-Object | Select-Object -First 10 | Remove-Item -Force
?? Funny thing is i tried to reproduce this today at home with no effect. Works as intended from here, even without parens.
Further I am painfully ignorant about how to use foreach statement, as it doesn't pipe the bastard :)
foreach ($file in $filepath) {$file} | Sort-Object | Select-Object -First 10 | Remove-Item -Force
Tried to put the sort and select-part in the {} too, but nothing good came of it. As i'm stuck in the pipe and don't understand the foreach logic.
your problem appears to be caused by how your pipeline works. [grin]
think about what it does ...
read ONE fileinfo item
send it to the pipeline
change/add a file
continue the pipeline
that 3rd step will cause the file list to change ... and may result in a file being read again OR some other change in the list of files to work on.
there are two solutions that come to mind ...
wrap the Get-ChildItem call in parens
that will force one read of the list before sending anything to the pipeline ... and that will ignore any changes caused by later pipeline stages.
use a foreach loop
that will read the whole list and then iterate thru the list one item at a time.
the 2nd solution also has the benefit of being easier to debug since the value of your current item only changes when explicitly modified. the current pipeline item changes at every pipeline stage ... and that is easy to forget. [grin]

using a delimiter to split files names in powershell

I use a simple function to download files and return the path to me when updating computers for simplicity.
I was stuck on why it was not working then realized that the proxy is appending a random number to the filename so instead of it being 12345.zip it is actually 8493830_12345.zip.
I have tried to find the file using the "_" as a split but while there are no errors, the file is not being returned and I have checked it is there manually.
function FileCheck {
$fileName.Split("_")[1]
$fileName = "{0}.zip" -f 12345
Download -ZipFileName $($fileName) -OutputDirectory $env:temp
$SleepTime = 300
$sleepElapsed = 0
$sleepInterval = 20
Start-Sleep $sleepInterval
$file = Get-ChildItem -Path $env:temp -Filter "$fileName*"
if ($file -ne $null) {
return $file[0].FullName
}
Start-Sleep($sleepInterval)
$sleepElapsed += $sleepInterval
if (($SleepTime) -le $sleepElapsed){
# Check for file with given prefix
$file = Get-ChildItem -Path $env:temp -Filter "$fileName*"
if ($file -eq $null) {
Write-Error 'file not found'
return $null
}
return $file[0].FullName
}
}
I am guessing the split is not working but googling and moving the filename.split has not worked for me. Any help is appreciated
Well, your split is doing nothing at all. You haven't defined $filename, but if you had, and it had an underscore, then $filename.split('_') would return two or more strings, depending on how many underscores were in the original string, but you never capture the result. I think the real problem here is the filter you are applying to Get-ChildItem later in your function.
$file = Get-ChildItem -Path $env:temp -Filter "$fileName*"
That will look for files beginning with $fileName, which you define on line 4 to be "12345.zip". That is exactly the opposite of what you want to be looking for. You need to move the asterisk to before $fileName, so it looks like this:
$file = Get-ChildItem -Path $env:temp -Filter "*$fileName"
That will return all files that end with "12345.zip", which would include things like:
myfuzzyhippo12345.zip
learn-to-count-12345.zip
8493830_12345.zip
Basically anything that ends in 12345.zip. Also, it appears that you are under the impression that executing a return $file[0].fullname or return $null will stop the function. That's a mistake. A function runs to completion unless exited early by something like a break command. Also, everything not explicitly captured or redirected will be passed back from the function, so reading through your function people are likely to get the output of your $filename.split('_') line, then possibly $null or $filename[0].fullname.
Lastly, it appears that you're trying to look for the file, if you don't find it to wait a bit, and try again, until $sleepElapsed is greater than $sleepTime. What you want here is a While or a Do/While loop. Here's what I'd do...
function FileCheck {
Param(
$fileName = '12345.zip',
$SleepTime = 300,
$sleepElapsed = 0,
$sleepInterval = 20
)
Download -ZipFileName $($fileName) -OutputDirectory $env:temp
Do{
Start-Sleep $sleepInterval
$sleepElapsed = $sleepElapsed + $sleepInterval
$file = Get-ChildItem -Path $env:temp -Filter "*$fileName"|Select -First 1
}While(!$file -and $sleepElapsed -le $sleepTime)
$file.FullName
}
That lets you define things like sleep settings at runtime if you want, or just let it default to what you were using, same with the file name. Then it downloads the file, and looks for it, pausing between attempts, until either it finds the file, or it runs out of time. Then it returns $file.FullName which is either the path to the file if it found one, or nothing if it didn't find a file.
Personally I'd have it return the file object, and just utilize the .FullName property if that's all I wanted later. Usually (not always, but usually) more info returned from a function is better than less info. Like what if the download fails and it's a zero byte file? Just returning only the path doesn't tell you that.

PS1 Replace or Remove a single character without rewriting the whole file

Is it possible to remove or replace the last character on the last non-whitespace line of a file using PowerShell 1?
I'm trying to get an Uptime log that is precise to within 5 minutes.
I've found that there are built logs and commands that can be accessed through command prompt that would tell me when the last time a computer was booted up, or when it shut down correctly, but the native uptime log only records once every 24 hrs, so if there is a power failure, I won't know how long the system has been offline with any precision more refined than 24 hours.
So I have created the following script:
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$uptime = (Get-Date)-$formBootTime
"$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00}" -f $uptime.Days,$uptime.Hours,$uptime.Minutes,$uptime.Seconds >> C:\UptimeTracker.csv
However, this gets tediously long to scroll through when I want to evaluate how long my machine has been running over the last X days.
So I thought I would add a marker to identify the current or most recent Uptime log per any given Boot.
But in order for that to work I would need to be able to remove said marker as soon as the previous record is no longer the relevant record.
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$file = (Get-Content c:\UptimeTracker.csv | Measure-Object)
$numberOfLines = $file.Count
$numberOfWords = (Get-Content c:\UptimeTracker.csv | Select -Index ($numberOfLines -1) | Measure-Object -word)
$Line = Get-Content c:\UptimeTracker.csv | Select -Index ($numberOfLines -2)
$wordArray = $Line.Split(",")
$LastLineBT = $wordArray[0]
if($LastLineBT -eq $formBootTime) {
$unmark = "true"
}
else
{$unmark = "false"}
if($unmark == "true"){ <remove last character of file> }
$uptime = (Get-Date)-$formBootTime
"$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00},X" -f $uptime.Days,$uptime.Hours,$uptime.Minutes, $uptime.Seconds >> C:\UptimeTracker.csv
Some of the above is borrowed and modified from: https://stackoverflow.com/a/16210970/11035837
I have seen several methods that receive the file as the input file and write to a different output file, and from there it would be an easy thing to do to script renaming the new and old files to switch their positions (new, old, standby - and rotate) the reason I'm trying not to rewrite the whole file is to reduce those instances where the command/script is interrupted and the action doesn't complete. Ideally the only time the action doesn't complete would be on a power failure. However, I have already seen in a previous version, it would skip 5 minute intervals occasionally for up to 15 minutes without any change in the last reported boot time. I suspect this has to do with other higher priority processes preventing the task scheduler from running the script. If this is the case, then a complete rewrite of the file failing part way through the script would lose some percentage of the existing log data, and I would rather miss the latest record than all the data.
Nothing I have found indicates any ability to remove/replace the last character (or two since one is a newline char), neither have I found anything that explicitly declares this is not possible - I have found declarations that it is not possible to elective replace inner or beginning content without a complete rewrite.
Barring any solution definitive answer, or if the definitive answer is no this cannot be done, then I will attempt something like the following:
if($unmark == "true"){
$input = "C:\UptimeTracker_CUR.csv"
$output = "C:\UptimeTracker_NEW.csv"
$content = Get-Content $input
$content[-2] = $content[-2] -replace 'X', ' '
$content | Set-Content $output
Rename-Item -Path "C:\UptimeTracker_CUR.csv" -NewName "C:\UptimeTracker_SBY.csv"
Rename-Item -Path "C:\UptimeTracker_NEW.csv" -NewName "C:\UptimeTracker_CUR.csv"
}
EDIT - due to multi-read comment by TheMadTechnician
...
$file = Get-Content c:\UptimeTracker.csv
$fileMeasure = ($file | Measure-Object)
$numberOfLines = $fileMeasure.Count
$numberOfWords = ($file | Select -Index ($numberOfLines -1) | Measure-Object -word)
$Line = $file | Select -Index ($numberOfLines -2)
...
...
if($unmark == "true"){
$output = "C:\UptimeTracker_NEW.csv"
$file[-2] = $file[-2] -replace 'X', ' '
$file | Set-Content $output
Rename-Item -Path "C:\UptimeTracker.csv" -NewName "C:\UptimeTracker_SBY.csv"
Rename-Item -Path "C:\UptimeTracker_NEW.csv" -NewName "C:\UptimeTracker.csv"
}
You read the whole file in several times, which has got to be slowing the whole script down. I would suggest reading the whole file in, determining if you need to clear your flag, then do so when you output, adding your new line to the file. Assuming you aren't still running PowerShell v2, you can do this:
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$uptime = (Get-Date)-$formBootTime
$File = Get-Content c:\UptimeTracker.csv -raw
if($File.trim().split("`n")[-1].Split(',')[0] -eq $formBootTime){
$File.trim() -replace 'X(?=\s*$)',' '
}else{
$File.Trim()
},("$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00},X" -f $uptime.Days,$uptime.Hours,$uptime.Minutes, $uptime.Seconds)|Set-Content c:\UptimeTracker.csv
If you are running an old version you will not have the -raw option for Get-Content. As a work around you can do this instead, and the same solution should still work.
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$uptime = (Get-Date)-$formBootTime
$File = (Get-Content c:\UptimeTracker.csv) -Join "`n"
if($File.trim().split("`n")[-1].Split(',')[0] -eq $formBootTime){
$File.trim() -replace 'X(?=\s*$)',' '
}else{
$File.Trim()
},("$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00},X" -f $uptime.Days,$uptime.Hours,$uptime.Minutes, $uptime.Seconds)|Set-Content c:\UptimeTracker.csv
This is going to be slower, so should be considered a secondary option since you'll have to read the whole file in as an array of strings, and convert it to a single multi-line string.

Powershell to Break up CSV by Number of Row

So I am now tasked with getting constant reports that are more than 1 Million lines long.
My last question did not explain all things so I'm tryin got do a better question.
I'm getting a dozen + daily reports that are coming in as CSV files. I don't know what the headers are or anything like that as I get them.
They are huge. I cant open in excel.
I wanted to basically break them up into the same report, just each report maybe 100,000 lines long.
The code I wrote below does not work as I keep getting a
Exception of type 'System.OutOfMemoryException' was thrown.
I am guessing I need a better way to do this.
I just need this file broken down to a more manageable size.
It does not matter how long it takes as I can run it over night.
I found this on the internet, and I tried to manipulate it, but I cant get it to work.
$PSScriptRoot
write-host $PSScriptRoot
$loc = $PSScriptRoot
$location = $loc
# how many rows per CSV?
$rowsMax = 10000;
# Get all CSV under current folder
$allCSVs = Get-ChildItem "$location\Split.csv"
# Read and split all of them
$allCSVs | ForEach-Object {
Write-Host $_.Name;
$content = Import-Csv "$location\Split.csv"
$insertLocation = ($_.Name.Length - 4);
for($i=1; $i -le $content.length ;$i+=$rowsMax){
$newName = $_.Name.Insert($insertLocation, "splitted_"+$i)
$content|select -first $i|select -last $rowsMax | convertto-csv -NoTypeInformation | % { $_ -replace '"', ""} | out-file $location\$newName -fo -en ascii
}
}
The key is not to read large files into memory in full, which is what you're doing by capturing the output from Import-Csv in a variable ($content = Import-Csv "$location\Split.csv").
That said, while using a single pipeline would solve your memory problem, performance will likely be poor, because you're converting from and back to CSV, which incurs a lot of overhead.
Even reading and writing the files as text with Get-Content and Set-Content is slow, however.
Therefore, I suggest a .NET-based approach for processing the files as text, which should substantially speed up processing.
The following code demonstrates this technique:
Get-ChildItem $PSScriptRoot/*.csv | ForEach-Object {
$csvFile = $_.FullName
# Construct a file-path template for the sequentially numbered chunk
# files; e.g., "...\file_split_001.csv"
$csvFileChunkTemplate = $csvFile -replace '(.+)\.(.+)', '$1_split_{0:000}.$2'
# Set how many lines make up a chunk.
$chunkLineCount = 10000
# Read the file lazily and save every chunk of $chunkLineCount
# lines to a new file.
$i = 0; $chunkNdx = 0
foreach ($line in [IO.File]::ReadLines($csvFile)) {
if ($i -eq 0) { ++$i; $header = $line; continue } # Save header line.
if ($i++ % $chunkLineCount -eq 1) { # Create new chunk file.
# Close previous file, if any.
if (++$chunkNdx -gt 1) { $fileWriter.Dispose() }
# Construct the file path for the next chunk, by
# instantiating the template with the next sequence number.
$csvFileChunk = $csvFileChunkTemplate -f $chunkNdx
Write-Verbose "Creating chunk: $csvFileChunk"
# Create the next chunk file and write the header.
$fileWriter = [IO.File]::CreateText($csvFileChunk)
$fileWriter.WriteLine($header)
}
# Write a data row to the current chunk file.
$fileWriter.WriteLine($line)
}
$fileWriter.Dispose() # Close the last file.
}
Note that the above code creates BOM-less UTF-8 files; if your input contains ASCII-range characters only, these files will effectively be ASCII files.
Here's the equivalent single-pipeline solution, which is likely to be substantially slower.
Get-ChildItem $PSScriptRoot/*.csv | ForEach-Object {
$csvFile = $_.FullName
# Construct a file-path template for the sequentially numbered chunk
# files; e.g., ".../file_split_001.csv"
$csvFileChunkTemplate = $csvFile -replace '(.+)\.(.+)', '$1_split_{0:000}.$2'
# Set how many lines make up a chunk.
$chunkLineCount = 10000
$i = 0; $chunkNdx = 0
Get-Content -LiteralPath $csvFile | ForEach-Object {
if ($i -eq 0) { ++$i; $header = $_; return } # Save header line.
if ($i++ % $chunkLineCount -eq 1) { #
# Construct the file path for the next chunk.
$csvFileChunk = $csvFileChunkTemplate -f ++$chunkNdx
Write-Verbose "Creating chunk: $csvFileChunk"
# Create the next chunk file and write the header.
Set-Content -Encoding ASCII -LiteralPath $csvFileChunk -Value $header
}
# Write data row to the current chunk file.
Add-Content -Encoding ASCII -LiteralPath $csvFileChunk -Value $_
}
}
Another option from linux world - split command. To get it on windows just install git bash, then you'll be able to use many linux tools in your CMD/powershell.
Below is the syntax to achieve your goal:
split -l 100000 --numeric-suffixes --suffix-length 3 --additional-suffix=.csv sourceFile.csv outputfile
It's very fast. If you want you can wrap split.exe as a cmdlet

Slow Powershell script for CSV modification

I'm using a powershell script to append data to the end of a bunch of files.
Each file is a CSV around 50Mb (Say 2 millionish lines), there are about 50 files.
The script I'm using looks like this:
$MyInvocation.MyCommand.path
$files = ls *.csv
foreach($f in $files)
{
$baseName = [System.IO.Path]::GetFileNameWithoutExtension($f)
$year = $basename.substring(0,4)
Write-Host "Starting" $Basename
$r = [IO.File]::OpenText($f)
while ($r.Peek() -ge 0) {
$line = $r.ReadLine()
$line + "," + $year | Add-Content $(".\DR_" + $basename + ".CSV")
}
$r.Dispose()
}
Problem is, it's pretty slow. It's taken about 12 hours to get through them.
It's not super complex, so I wouldn't expect it to take that long to run.
What could I do to speed it up?
Reading and writing a file row by row can be a bit slow. Maybe your antivirus is contributing to slowness as well. Use Measure-Command to see which parts of the script are the slow ones.
As a general advise, rather write a few large blocks instead of lots of small ones. You can achieve this by storing some content in a StringBuilder and appending its contents into the output file every, say, 1000 processed rows. Like so,
$sb = new-object Text.StringBuilder # New String Builder for stuff
$i = 1 # Row counter
while ($r.Peek() -ge 0) {
# Add formatted stuff into the buffer
[void]$sb.Append($("{0},{1}{2}" -f $r.ReadLine(), $year, [Environment]::NewLine ) )
if(++$i % 1000 -eq 0){ # When 1000 rows are added, dump contents into file
Add-Content $(".\DR_" + $basename + ".CSV") $sb.ToString()
$sb = new-object Text.StringBuilder # Reset the StringBuilder
}
}
# Don't miss the tail of the contents
Add-Content $(".\DR_" + $basename + ".CSV") $sb.ToString()
Don't go into .NET Framework static methods and building up strings when there are cmdlets that can do the work on objects. Collect your data, add the year column, then export to your new file. You're also doing a ton of file I/O and that'll also slow you down.
This will probably require a little bit more memory. But it reads the whole file at once, and writes the whole file at once. It also assumes that your CSV files have column headings. But it's much easier for someone else to look at and understand exactly what's going on (write your scripts so they can be read!).
# Always use full cmdlet names in scripts, not aliases
$files = get-childitem *.csv;
foreach($f in $files)
{
#basename is a property of the file object in PowerShell, there's no need to call a static method
$basename = $f.basename;
$year = $f.basename.substring(0,4)
# Every time you use Write-Host, a puppy dies
"Starting $Basename";
# If you've got CSV data, treat it as CSV data. PowerShell can import it into a collection natively.
$data = Import-Csv $f;
$exportData = #();
foreach ($row in $data) {
# Add a year "property" to each row object
$row |Add-Member -membertype NoteProperty -Name "Year" -Value $year;
# Export the modified row to the output file
$row |Export-Csv -NoTypeInformation -Path $("r:\DR_" + $basename + ".CSV") -Append -NoClobber
}
}