Change and save .nc files - powershell

I have a massive amount of .nc files (text files) where I need to change different lines based on their linenumer and content.
Example:
So far I have:
Get-ChildItem I:\temp *.nc -recurse | ForEach-Object {
$c = ($_ | Get-Content)
$c = $c -replace "S355J2","S235JR2"
$c = $c.GetType() | Format-Table -AutoSize
$c = $c -replace $c[3],$c[4]
[IO.File]::WriteAllText($_.FullName, ($c -join "`r`n"))
}
This is not working, however, since it returns only a few PowerShell lines to each file, instead of the original (changed) content.

I don't know what you expect $c = $c.GetType() | Format-Table -AutoSize to do, but it most likely doesn't do whatever it is you're expecting.
If I understand your question correctly you essentially want to
remove the line pos,
replace the code S355J2 with S235JR2, and
remove a section SI if it exists.
The following code should work:
Get-ChildItem I:\temp *.nc -Recurse | ForEach-Object {
(Get-Content $_.FullName | Out-String) -replace 'pos\r\n\s+' -replace 'S355J2', 'S235JR2' -replace '(?m)^SI\r\n(\s+.*\n)+' |
Set-Content $_.FullName
}
Out-String mangles the content of the input file into a single string, and the daisy-chained replacement operations modify that string before it's written back to the file. The expression (?m)^SI\r\n(\s+.*\n)+ matches a line beginning with SI and followed by one or more indented lines. The (?m) modifier is to allow matching start-of-line in a multiline string, otherwise ^ would only match the beginning of the string.
Edit: If you need to replace variable text in the 3rd line with the text from the 4th line (thus duplicating the 4th line) you're indeed better off working with an array for that. Delay the mangling of the string array until after that replacement:
Get-ChildItem I:\temp *.nc -Recurse | ForEach-Object {
$txt = #(Get-Content $_.FullName)
$txt[3] = $txt[4]
($txt | Out-String) -replace 'S355J2', 'S235JR2' -replace '(?m)^SI\r\n(\s+.*\n)+' |
Set-Content $_.FullName
}

Related

How to strip out leading time stamp?

I have some log files.
Some of the UPDATE SQL statements are getting errors, but not all.
I need to know all the statements that are getting errors so I can find the pattern of failure.
I can sort all the log files and get the unique lines, like this:
$In = "C:\temp\data"
$Out1 = "C:\temp\output1"
$Out2 = "C:\temp\output2"
Remove-Item $Out1\*.*
Remove-Item $Out2\*.*
# Get the log files from the last 90 days
Get-ChildItem $In -Filter *.log | Where-Object {$_.LastWriteTime -gt (Get-Date).AddDays(-90)} |
Foreach-Object {
$content = Get-Content $_.FullName
#filter and save content to a file
$content | Where-Object {$_ -match 'STATEMENT'} | Sort-Object -Unique | Set-Content $Out1\$_
}
# merge all the files, sort unique, write to output
Get-Content $Out2\* | Sort-Object -Unique | Set-Content $Out3\output.txt
Works great.
But some of the logs have a leading date-time stamp in the leading 24 char. I need to strip that out, or all those lines are unique.
If it helps, all the files either have the leading timestamp or they don't. The lines are not mixed within a single file.
Here is what I have so far:
# Get the log files from the last 90 days
Get-ChildItem $In -Filter *.log | Where-Object {$_.LastWriteTime -gt (Get-Date).AddDays(-90)} |
Foreach-Object {
$content = Get-Content $_.FullName
#filter and save content to a file
$s = $content | Where-Object {$_ -match 'STATEMENT'}
# strip datetime from front if exists
If (Where-Object {$s.Substring(0,1) -Match '/d'}) { $s = $s.Substring(24) }
$s | Sort-Object -Unique | Set-Content $Out1\$_
}
# merge all the files, sort unique, write to output
Get-Content $Out1\* | Sort-Object -Unique | Set-Content $Out2\output.txt
But it just write the lines out without stripping the leading chars.
Regex /d should be \d (\ is the escape character in general, and character-class shortcuts such as d for a digit[1] must be prefixed with it).
Use a single pipeline that passes the Where-Object output to a ForEach-Object call where you can perform the conditional removal of the numeric prefix.
$content |
Where-Object { $_ -match 'STATEMENT' } |
ForEach-Object { if ($_[0] -match '\d') { $_.Substring(24) } else { $_ } } |
Set-Content $Out1\$_
Note: Strictly speaking, \d matches everything that the Unicode standard considers a digit, not just the ASCII-range digits 0 to 9; to limit matching to the latter, use [0-9].

Why does this code only output lines that I made a replacement on?

I wrote this code to test out text manipulation. For each line read from my text file I replace tabs/returns/spaces, then I check if the line contains the carachters 'SAAS' and remove an A. I then write the content to an new file.
The issue is that the new file contains only lines that I made a replacement on and deletes any other lines of text from the orinal file when writing to the new file.
$Text = Get-Content -Path C:\Desktop\Phones\Phones_1.txt |
ForEach-Object {($_ -replace '\n','')} |
ForEach-Object {($_ -replace '\r','')} |
ForEach-Object {($_ -replace '\s','')} |
ForEach-Object {IF($_ | Select-String -Pattern 'SAAS'){$_ -replace 'SAAS','SAS'}}
$Text | Out-File 'C:\Desktop\Phones\phone2.txt'
Any help is appriciated.
It probably has to do with your Select-String function, which I don't believe needs to be there. Also, you can chain together replace statements greatly reducing the need to keep piping your code. I don't think it causes any issues, but you don't need to wrap your ForEach-Object blocks in braces either. Here's what that would look like:
$Text = Get-Content -Path C:\Desktop\Phones\Phones_1.txt |
ForEach-Object { $_ -replace '\n','' -replace '\r','' -replace '\s','' -replace 'SAAS','SAS' }
$Text | Out-File 'C:\Desktop\Phones\phone2.txt'
As $Text contains an array of single lines, replacing \r,\n is pretty useless.
You should provide an example of your input and expected output by editing your question.
Using a RegularExpression with lookarounds
(Get-Content .\Phones_1.txt -raw) -replace '\r|\n|\s|(?<=SA)A(?=S)'|Set-Content Phone2.txt
Yields this output from your above questions complete text:
Iwrotethiscodetotestouttextmanipulation.ForeachlinereadfrommytextfileIreplacetabs/returns/spaces,thenIcheckifthelinecontainsthecarachters'SAS'andremoveanA.Ithenwritethecontenttoannewfile.TheissueisthatthenewfilecontainsonlylinesthatImadeareplacementonanddeletesanyotherlinesoftextfromtheorinalfilewhenwritingtothenewfile.$Text=Get-Content-PathC:\Desktop\Phones\Phones_1.txt|ForEach-Object{($-replace'\n','')}|ForEach-Object{($-replace'\r','')}|ForEach-Object{($-replace'\s','')}|ForEach-Object{IF($|Select-String-Pattern'SAS'){$_-replace'SAS','SAS'}}$Text|Out-File'C:\Desktop\Phones\phone2.txt'Anyhelpisappriciated.

Need to output multiple rows to CSV file

I am using the following script that iterates through hundreds of text files looking for specific instances of the regex expression within. I need to add a second data point to the array, which tells me the object the pattern matched in.
In the below script the [Regex]::Matches($str, $Pattern) | % { $_.Value } piece returns multiple rows per file, which cannot be easily output to a file.
What I would like to know is, how would I output a 2 column CSV file, one column with the file name (which should be $_.FullName), and one column with the regex results? The code of where I am at now is below.
$FolderPath = "C:\Test"
$Pattern = "(?i)(?<=\b^test\b)\s+(\w+)\S+"
$Lines = #()
Get-ChildItem -Recurse $FolderPath -File | ForEach-Object {
$_.FullName
$str = Get-Content $_.FullName
$Lines += [Regex]::Matches($str, $Pattern) |
% { $_.Value } |
Sort-Object |
Get-Unique
}
$Lines = $Lines.Trim().ToUpper() -replace '[\r\n]+', ' ' -replace ";", '' |
Sort-Object |
Get-Unique # Cleaning up data in array
I can think of two ways but the simplest way is to use a hashtable (dict). Another way is create psobjects to fill your Lines variable. I am going to go with the simple way so you can only use one variable, the hashtable.
$FolderPath = "C:\Test"
$Pattern = "(?i)(?<=\b^test\b)\s+(\w+)\S+"
$Results =#{}
Get-ChildItem -Recurse $FolderPath -File |
ForEach-Object {
$str = Get-Content $_.FullName
$Line = [regex]::matches($str,$Pattern) | % { $_.Value } | Sort-Object | Get-Unique
$Line = $Line.Trim().ToUpper() -Replace '[\r\n]+', ' ' -Replace ";",'' | Sort-Object | Get-Unique # Cleaning up data in array
$Results[$_.FullName] = $Line
}
$Results.GetEnumerator() | Select #{L="Folder";E={$_.Key}}, #{L="Matches";E={$_.Value}} | Export-Csv -NoType -Path <Path to save CSV>
Your results will be in $Results. $Result.keys contain the folder names. $Results.Values has the results from expression. You can reference the results of a particular folder by its key $Results["Folder path"]. of course it will error if the key does not exist.

Filtering sections of data including the starting and ending lines- PowerShell

I have a text file that looks like this:
Data I'm NOT looking for
More data that doesn't matter
Even more data that I don't
&Start/Finally the data I'm looking for
&Data/More data that I need
&Stop/I need this too
&Start/Second batch of data I need
&Data/I need this too
&Stop/Okay now I'm done
Ending that I don't need
Here is what the output needs to be:
File1.txt
&Start/Finally the data I'm looking for
&Data/More data that I need
&Stop/I need this too
File2.txt
&Start/Second batch of data I need
&Data/I need this too
&Stop/Okay now I'm done
I need to do this for every file in a folder (sometimes there will be multiple files that will need to be filtered.) The files names can be incrementing: ex. File1.txt, File2.txt, File3.txt.
This is what I have tried with no luck:
ForEach-Object{
$text -join "`n" -split '(?ms)(?=^&START)' -match '^&START' |
Out-File B:\PowerShell\$filename}
Thanks!
Looks like you were pretty close: your code correctly extracted the paragraphs of interest, but intra-paragraph out-filtering of non-&-starting lines was missing, and you needed to write to paragraph-specific output files:
$text -join "`n" -split '(?m)(?=^&Start)' -match '^&Start' |
ForEach-Object { $ndx=0 } { $_ -split '\n' -match '^&' | Out-File "File$((++$ndx)).txt" }
This creates sequentially numbered files starting with File1.txt for every paragraph of interest.
To do it for every file in a folder, with output filenames using fixed naming scheme File<n> across all input files (and thus cumulative numbering):
Get-ChildItem -File . | ForEach-Object -Begin { $ndx=0 } -Process {
(Get-Content -Raw $_) -split '(?m)(?=^&Start)' -match '^&Start' |
ForEach-Object { $_ -split '\n' -match '^&' | Out-File "File$((++$ndx)).txt" }
}
To do it for every file in a folder, with output filenames based on the input filenames and numbering per input file (PSv4+, due to use of -PipelineVariable):
Get-ChildItem -File . -PipelineVariable File | ForEach-Object {
(Get-Content -Raw $_) -split '(?m)(?=^&Start)' -match '^&Start' |
ForEach-Object {$ndx=0} { $_ -split '\n' -match '^&' | Out-File "$($File.Name)$((++$ndx)).txt" }
}
You post a second question (against the rules) and it was deleted but here is my quick answer for it. I hope it will help you and give you more sense how PS works:
$InputFile = "C:\temp\test\New folder (3)\File1.txt"
# get file content
$a=Get-Content $InputFile
# loop for every line in range 2 to last but one
for ($i=1; $i -lt ($a.count-1); $i++)
{
#geting string part between & and / , and construct output file name
$OutFile = "$(Split-Path $InputFile)\$(($a[$i] -split '/')[0] -replace '&','').txt"
$a[0]| Out-File $OutFile #creating output file and write first line in it
$a[$i]| Out-File $OutFile -Append #write info line
$a[-1]| Out-File $OutFile -Append #write last line
}
Something like this?
$i=0
gci -path "C:\temp\ExplodeDir" -file | %{ (get-content -path $_.FullName -Raw).Replace("`r`n`r`n", ";").Replace("`r`n", "~").Split(";") | %{if ($_ -like "*Start*") {$i++; ($_ -split "~") | out-file "C:\temp\ResultFile\File$i.txt" }} }

Powershell: addin line into the .txt file

I have a text (.txt) file with following content:
Car1
Car2
Car3
Car4
Car5
For changing Car1 for random text I used this script:
Get-ChildItem "C:\Users\boris.magdic\Desktop\q" -Filter *.TXT |
Foreach-Object{
$content = Get-Content $_.FullName
$content | ForEach-Object { $_ -replace "Car1", "random_text" } | Set-Content $_.FullName
}
This is working ok, but now I want to add one text line under Car2 in my text file.
How can I do that?
Just chain another -replace and use a new line!
Get-ChildItem "C:\Users\boris.magdic\Desktop\q" -Filter *.TXT |
Foreach-Object{
$file = $_.FullName
$content = Get-Content $file
$content | ForEach-Object { $_ -replace "Car1", "random_text" -replace "(Car2)","`$1`r`nOtherText" } | Set-Content $file
}
First thing is that | Set-Content $_.FullName would not work since the file object does not exist in that pipe. So one simple this to do it save the variable for use later in the pipe. You can also use the ForEach($file in (Get-ChildItem....)) construct.
The specific change to get what you want is the second -replace. We place what you want to match in brackets to that we can reference it in the replacement string with $1. We use a backtick to ensure PowerShell does not treat it as a variable.
We can remove some redundancy as well since -replace will work against the strings of file as a whole
Get-ChildItem "c:\temp" -Filter *.TXT |
Foreach-Object{
$file = $_.FullName
(Get-Content $file) -replace "Car1", "random_text" -replace "(Car2)","`$1`r`nOtherText" | Set-Content $file
}
While this does work with your sample text I want to point out that more complicated strings might require more finesse to ensure you make the correct changed and that the replacements we are using are regex based and do not need to be for this specific example.
.Replace()
So if you were just doing simple replacements then we can update your original logic.
Foreach-Object{
$file = $_.FullName
$content = Get-Content $_.FullName
$content | ForEach-Object { $_.replace("Car1", "random_text").replace("Car2","Car2`r`nOtherText")} | Set-Content $file
}
So that is just simple text replacement chained using the string method .Replace()