I would like to get content from files in a folder (ignoring the header lines, since some file may ONLY contain the header). But in the output, I would like to include the filename from which the line is read. So far, I have the following:
Get-ChildItem | Get-Content | Where { $_ -notlike "HEADER_LINE_TEXT" } | Out-File -FilePath output_text.txt
I've tried to work with creating a variable in the Where block, $filename=$_.BaseName, and using it in the output, but this didn't work.
EDIT:
I ended up with the following:
Get-ChildItem -Path . |
Where-Object { $_.FullName -like "*records.txt"; $fname=$_FullName; } |
Get-Content |
Select-Object { ($fname + "|" + $_.Trim()) } |
Where { $_ -notlike "*HEADER_LINE_TEXT*" } |
Format-Table -HideTableHeaders |
Out-File -FilePath output_text.txt
This looks lengthy, and can probably be made shorter and clearer. Can someone help with cleaning this up a bit? I'll either post the solution, or vote for a cleaner solution, if one is posted. Thanks.
This looks like a case where it would make it more readable to not make it a one liner at cost of a little additional memory usage.
$InputFolder = "C:\example"
$OutputFile = "C:\example\output_text.txt"
$Files = Get-ChildItem $InputFolder | Where-Object { $_.FullName -like "*records.txt"}
Foreach ($File in $Files) {
$FilteredContent = Get-Content $File.FullName | Where-Object {$_ -notlike "*HEADER_LINE_TEXT*"}
$Output = $FilteredContent | Foreach-Object { "$($File.FullName)|$($_.Trim())" }
$Output | Out-File $OutputFile -Append
}
If you are going to go oneliner style for brevity, you could cut down on length by using position for parameters and using aliases.
Here are a couple other changes:
No need for the second semicolon in your first where block.
I think your variable wasn't working because you were missing the period between $_ and fullname.
Format-Table isn't needed because you already have the string you want to output
You can optimize a little by moving the second where earlier so that you don't trim() on lines you are just going to filter
Looks like you want to use foreach instead of select
Removed the + operator for string concatenation, instead using $() to evaluate inside parenthesis
gci . |
? { $_.FullName -like "*records.txt"; $fname=$_.FullName } |
% { gc $_.FullName } |
? { $_ -notlike "*HEADER_LINE_TEXT*" } |
% { "$fname|$($_.Trim())" } |
Out-File output_text.txt
Related
I have some log files.
Some of the UPDATE SQL statements are getting errors, but not all.
I need to know all the statements that are getting errors so I can find the pattern of failure.
I can sort all the log files and get the unique lines, like this:
$In = "C:\temp\data"
$Out1 = "C:\temp\output1"
$Out2 = "C:\temp\output2"
Remove-Item $Out1\*.*
Remove-Item $Out2\*.*
# Get the log files from the last 90 days
Get-ChildItem $In -Filter *.log | Where-Object {$_.LastWriteTime -gt (Get-Date).AddDays(-90)} |
Foreach-Object {
$content = Get-Content $_.FullName
#filter and save content to a file
$content | Where-Object {$_ -match 'STATEMENT'} | Sort-Object -Unique | Set-Content $Out1\$_
}
# merge all the files, sort unique, write to output
Get-Content $Out2\* | Sort-Object -Unique | Set-Content $Out3\output.txt
Works great.
But some of the logs have a leading date-time stamp in the leading 24 char. I need to strip that out, or all those lines are unique.
If it helps, all the files either have the leading timestamp or they don't. The lines are not mixed within a single file.
Here is what I have so far:
# Get the log files from the last 90 days
Get-ChildItem $In -Filter *.log | Where-Object {$_.LastWriteTime -gt (Get-Date).AddDays(-90)} |
Foreach-Object {
$content = Get-Content $_.FullName
#filter and save content to a file
$s = $content | Where-Object {$_ -match 'STATEMENT'}
# strip datetime from front if exists
If (Where-Object {$s.Substring(0,1) -Match '/d'}) { $s = $s.Substring(24) }
$s | Sort-Object -Unique | Set-Content $Out1\$_
}
# merge all the files, sort unique, write to output
Get-Content $Out1\* | Sort-Object -Unique | Set-Content $Out2\output.txt
But it just write the lines out without stripping the leading chars.
Regex /d should be \d (\ is the escape character in general, and character-class shortcuts such as d for a digit[1] must be prefixed with it).
Use a single pipeline that passes the Where-Object output to a ForEach-Object call where you can perform the conditional removal of the numeric prefix.
$content |
Where-Object { $_ -match 'STATEMENT' } |
ForEach-Object { if ($_[0] -match '\d') { $_.Substring(24) } else { $_ } } |
Set-Content $Out1\$_
Note: Strictly speaking, \d matches everything that the Unicode standard considers a digit, not just the ASCII-range digits 0 to 9; to limit matching to the latter, use [0-9].
I'm again stuck on something that should be so simple. I have a CSV file in which I need to do a few string modifications and export it back out. The data looks like this:
FullName
--------
\\server\project\AOI
\\server\project\AOI\Folder1
\\server\project\AOI\Folder2
\\server\project\AOI\Folder3\User
I need to do the following:
Remove the "\\server\project" from each line but leave the rest of the line
Delete all rows which do not have a Folder (e.g., in the example above, the first row would be deleted but the other three would remain)
Delete any row with the word "User" in the path
Add a column called T/F with a value of "FALSE" for each record
Here is my initial attempt at this:
Get-Content C:\Folders.csv |
% {$_.replace('\\server\project\','')} |
Where-Object {$_ -match '\\'} |
#Removes User Folders rows from CSV
Where-Object {$_ -notmatch 'User'} |
Out-File C:\Folders-mod.csv
This works to a certain extent, except it deletes my header row and I have not found a way to add a column using Get-Content. For that, I have to use Import-Csv, which is fine, but it seems inefficient to be constantly reloading the same file. So I tried rewriting the above using Import-Csv instead of Get-Content:
$Folders = Import-Csv C:\Folders.csv
foreach ($Folder in $Folders) {
$Folder.FullName = $Folder.FullName.Replace('\\server\AOI\', '') |
Where-Object {$_ -match '\\'} |
Where-Object {$_ -notmatch 'User Files'}
}
$Folders | Export-Csv C:\Folders-mod.csv -NoTypeInformation
I haven't added the coding for adding the new column yet, but this keeps the header. However, I end up with a bunch of empty rows where the Where-Object deletes the line, and the only way I can find to get rid of them is to run the output file through a Get-Content command. This all seems overly complicated for something that should be simple.
So, what am I missing?
Thanks to TheMadTechnician for pointing out what I was doing wrong. Here is my final script (with additional column added):
$Folders= Import-CSV C:\Folders.csv
ForEach ($Folder in $Folders)
{
$Folder.FullName = $Folder.FullName.replace('\\server\project\','')
}
$Folders | Where-Object {$_ -match '\\' -and $_ -notmatch 'User'} |
Select-Object *,#{Name='T/F';Expression={'FALSE'}} |
Export-CSV C:\Folders.csv -NoTypeInformation
I would do this with a Table Array and pscustomobject.
#Create an empty Array
$Table = #()
#Manipulate the data
$Fullname = Get-Content C:\Folders.csv |
ForEach-Object {$_.replace('\\server\project\', '')} |
Where-Object {$_ -match '\\'} |
#Removes User Folders rows from CSV
Where-Object {$_ -notmatch 'User'}
#Define custom objects
Foreach ($name in $Fullname) {
$Table += [pscustomobject]#{'Fullname' = $name; 'T/F' = 'FALSE'}
}
#Export results to new csv
$Table | Export-CSV C:\Folders-mod.csv -NoTypeInformation
here's yet another way to do it ... [grin]
$FileList = #'
FullName
\\server\project\AOI
\\server\project\AOI\Folder1
\\server\project\AOI\Folder2
\\server\project\AOI\Folder3\User
'# | ConvertFrom-Csv
$ThingToRemove = '\\server\project'
$FileList |
Where-Object {
# toss out any blank lines
$_ -and
# toss out any lines with "user" in them
$_ -notmatch 'User'
} |
ForEach-Object {
[PSCustomObject]#{
FullName = $_.FullName -replace [regex]::Escape($ThingToRemove)
'T/F' = $False
}
}
output ...
FullName T/F
-------- ---
\AOI False
\AOI\Folder1 False
\AOI\Folder2 False
notes ...
putting a slash in the property name is ... icky [grin]
that requires wrapping the property name in quotes every time you need to access it. try another name - perhaps "Correct".
you can test for blank array items [lines] with $_ all on its own
the [regex]::Escape() stuff is really quite handy
We have a file server that processes files that are received. When a file fails to process for whatever reason, it is moved into a failure folder. I've written a script to iterate through every possible one of these folders and spit out the FullName of the file into an e-mail which it sends to me.
Now when I run it manually, it works fine. However, when I set it as a scheduled task (running as Local System), the script still runs successfully, but the e-mail contains paths like \\blahblah\blah\blahblahblah\bl.....
I've tweaked the script a bunch of different ways and every time the output ends up the same. When I run it manually, it works as intended, when it runs as an automated script, it truncates the FullNames. I've found other people with this issue, but not as an automated task.
This is the relevant code of the script.
$emailFileList = ""
$filelist = #()
try {
GCI $topLevelPath -Recurse |
? { $_.PSIsContainer } |
ForEach-Object {
dir $_.FullName |
Where-Object {$_.FullName -like $unableToProcess} | ForEach-Object {
$filelist += dir $_.FullName
}
}
$emailFileList = Out-String -InputObject $($filelist | Select-Object FullName | Format-Table -AutoSize)
$emailBody = $emailBody + $emailFileList
}
EDIT:
I used the HTML method below but it added a bunch of junk markup. I added 4 lines to replace the markup with quotes. The inside of the try block now looks like this, and it works even as scheduled tasks.
GCI $topLevelPath -Recurse |
? { $_.PSIsContainer } |
ForEach-Object {
dir $_.FullName |
Where-Object {$_.FullName -like $unableToProcess} | ForEach-Object {
$filelist += dir $_.FullName
}
}
$emailFileList = $filelist | Select-Object FullName | ConvertTo-Html -fragment
$emailFileList = [regex]::Replace($emailFileList, "<table>.+</th></tr>", "")
$emailFileList = $emailFileList -replace '<tr><td>', '"'
$emailFileList = $emailFileList -replace '</td></tr>', """`r`n"
$emailFileList = $emailFileList -replace '</table>', ''
$emailBody = $emailBody + $emailFileList
I guess I also technically used regex on html what have I done noooooooo
Edit: Regardling answer "duplication" the problem above is SPECIFICALLY an interaction between powershell and the windows scheduled tasks.
This gives the kind of output you would probably expect
[command] | Format-Table -AutoSize | Out-String -Width 10000 #| clip.exe
Since you were using Format-Table -Autosize it was probably truncating due to the the amount of characters per line in the powershell instance. You can use the ConvertTo-Html function with the -Fragment command to create an HTML table.
Try something like this:
$emailFileList = ""
$filelist = #()
try {
Get-ChildItem $topLevelPath -Recurse `
| Where-Object -Property PSIsContainer -EQ -Value $True `
| ForEach-Object {
Get-ChildItem $_.FullName |
Where-Object -Property FullName -Like -Value $UnableToProcess `
| ForEach-Object {
$filelist += Get-ChildItem $_.FullName
}
}
$emailFileList = $filelist | Select-Object FullName | ConvertTo-Html -Fragment
$emailBody = $emailBody + $emailFileList
}
catch
{
}
Your problem is the formatter truncating because the console host can't render the full strings. Here's a solution where you'll get a txt with a list of names that can be used however you want
General rule of thumb: filter left, format right.
#Requires -Version 3
Try
{
Get-ChildItem -Path $TopLevelPath -Recurse -Folder |
## If you're on version 2, replace the -Folder switch with the following:
#Where-Object { $_.PSIsContainer }
ForEach-Object {
## If version 2, remove #().FullName and replace with
# | Select-Object -ExpandProperty FullName
$FileList = #(Get-ChildItem -Path $_.FullName -Filter "*$UnableToProcess*").FullName
}
$EmailBody += #($FileList)
}
Catch
{
}
I have a PowerShell script that I use to change text in a number of files. The following script will work & changes the text as expected.
Get-ChildItem $FileFolder -Recurse |
select -ExpandProperty fullname |
foreach {
(Get-Content $_) |
ForEach-Object {$_ -replace $old $new } |
Set-Content $_
}
The problem is though, it changes every file that it opens, so everything has a timestamp of when the job was run even if nothing was changed.
I have tried something similar to what is here but it gives me an error:
The term 'if' is not recognized as the name of a cmdlet, function, etc...
Here is the code I am trying to run:
Get-ChildItem $FileFolder -Recurse |
select -ExpandProperty fullname |
foreach {
$b = ($a = Get-Content $_) |
ForEach-Object {$_ -replace $old $new } |
if (Compare $a $b -PassThru) {
$b | Set-Content $_
}
}
I know that the code isn't right, but if I move it inside the ForEach-Object, it won't run either.
What I want to do is to use the Set-Content statement only if the contents of the file have changed. Would appreciate any thoughts as to how best to do this.
What you can do is look for the string before getting and setting content. Something like:
Get-ChildItem $FileFolder -Recurse |
select -ExpandProperty fullname |
foreach {
If(Select-String -Path $_ -SimpleMatch $old -quiet){
(Get-Content $_) |
ForEach-Object {$_ -replace $old $new } |
Set-Content $_
}
}
I run the following code using PowerShell to get a list of add/remove programs from the registry:
Get-ChildItem -path hklm:\software\microsoft\windows\currentversion\uninstall `
| ForEach-Object -Process { Write-Output $_.GetValue("DisplayName") } `
| Out-File addrem.txt
I want the list to be separated by newlines per each program. I've tried:
Get-ChildItem -path hklm:\software\microsoft\windows\currentversion\uninstall `
| ForEach-Object -Process { Write-Output $_.GetValue("DisplayName") `n } `
| out-file test.txt
Get-ChildItem -path hklm:\software\microsoft\windows\currentversion\uninstall `
| ForEach-Object {$_.GetValue("DisplayName") } `
| Write-Host -Separator `n
Get-ChildItem -path hklm:\software\microsoft\windows\currentversion\uninstall `
| ForEach-Object -Process { $_.GetValue("DisplayName") } `
| foreach($_) { echo $_ `n }
But all result in weird formatting when output to the console, and with three square characters after each line when output to a file. I tried Format-List, Format-Table, and Format-Wide with no luck. Originally, I thought something like this would work:
Get-ChildItem -path hklm:\software\microsoft\windows\currentversion\uninstall `
| ForEach-Object -Process { "$_.GetValue("DisplayName") `n" }
But that just gave me an error.
Or, just set the output field separator (OFS) to double newlines, and then make sure you get a string when you send it to file:
$OFS = "`r`n`r`n"
"$( gci -path hklm:\software\microsoft\windows\currentversion\uninstall |
ForEach-Object -Process { write-output $_.GetValue('DisplayName') } )" |
out-file addrem.txt
Beware to use the ` and not the '. On my keyboard (US-English Qwerty layout) it's located left of the 1.
(Moved here from the comments - Thanks Koen Zomers)
Give this a try:
PS> $nl = [Environment]::NewLine
PS> gci hklm:\software\microsoft\windows\currentversion\uninstall |
ForEach { $_.GetValue("DisplayName") } | Where {$_} | Sort |
Foreach {"$_$nl"} | Out-File addrem.txt -Enc ascii
It yields the following text in my addrem.txt file:
Adobe AIR
Adobe Flash Player 10 ActiveX
...
Note: on my system, GetValue("DisplayName") returns null for some entries, so I filter those out. BTW, you were close with this:
ForEach-Object -Process { "$_.GetValue("DisplayName") `n" }
Except that within a string, if you need to access a property of a variable, that is, "evaluate an expression", then you need to use subexpression syntax like so:
Foreach-Object -Process { "$($_.GetValue('DisplayName'))`r`n" }
Essentially within a double quoted string PowerShell will expand variables like $_, but it won't evaluate expressions unless you put the expression within a subexpression using this syntax:
$(`<Multiple statements can go in here`>).
I think you had the correct idea with your last example. You only got an error because you were trying to put quotes inside an already quoted string. This will fix it:
gci -path hklm:\software\microsoft\windows\currentversion\uninstall | ForEach-Object -Process { write-output ($_.GetValue("DisplayName") + "`n") }
Edit: Keith's $() operator actually creates a better syntax (I always forget about this one). You can also escape quotes inside quotes as so:
gci -path hklm:\software\microsoft\windows\currentversion\uninstall | ForEach-Object -Process { write-output "$($_.GetValue(`"DisplayName`"))`n" }
Ultimately, what you're trying to do with the EXTRA blank lines between each one is a little confusing :)
I think what you really want to do is use Get-ItemProperty. You'll get errors when values are missing, but you can suppress them with -ErrorAction 0 or just leave them as reminders. Because the Registry provider returns extra properties, you'll want to stick in a Select-Object that uses the same properties as the Get-Properties.
Then if you want each property on a line with a blank line between, use Format-List (otherwise, use Format-Table to get one per line).
gci -path hklm:\software\microsoft\windows\currentversion\uninstall |
gp -Name DisplayName, InstallDate |
select DisplayName, InstallDate |
fl | out-file addrem.txt
The option that I tend to use, mostly because it's simple and I don't have to think, is using Write-Output as below. Write-Output will put an EOL marker in the string for you and you can simply output the finished string.
Write-Output $stringThatNeedsEOLMarker | Out-File -FilePath PathToFile -Append
Alternatively, you could also just build the entire string using Write-Output and then push the finished string into Out-File.