Iterating over multiple find and replace object - powershell

I know the solution below is quite simple but I am not able to get it to work. I created an object with a find and replace string property. The goal to replace all strings in the text file but it is not working properly. The code below would only replace the last object in the list. I've searched several stack over flow questions but was only able to find multiple files, not multiple find/replace strings.
Input File:
TEMPERATURE 1
TEMPERATURE 2
AVERAGE 1
AVERAGE 2
Expected Output:
TEMP 1
TEMP 2
AVG 1
AVG 2
Actual Output:
TEMP 1
TEMP 2
AVERAGE 1
AVERAGE 2
class FindReplace {
[string]$FindString;
[string]$ReplaceString;
}
[System.Collections.Generic.List[FindReplace]]$FindReplaceList = #()
$Obj1 = New-Object FindReplace
$Obj1.FindString = "AVERAGE"
$Obj1.ReplaceString = "AVG"
$Obj2 = New-Object FindReplace
$Obj2.FindString = "TEMPERATURE"
$Obj2.ReplaceString = "TEMP"
$FindReplaceList.Add($Obj1);
$FindReplaceList.Add($Obj2);
write-host $FindReplaceList.Count
for($i = 0; $i -lt $FindReplaceList.Count; $i++)
{
Write-Host "Replacing string" $FindReplaceList[$i].FindString "to" $FindReplaceList[$i].ReplaceString " where input file:" $inputTextFilePath "and output file:" $outputTextFilePath
(Get-Content $inputTextFilePath).Replace($FindReplaceList[$i].FindString, $FindReplaceList[$i].ReplaceString) | Set-Content $outputTextFilePath
}

As #AdminOfThings mentioned in the comments. The problem is that you are reading the file once, and then writing twice. So you don't end up with all of the replacements like you are expecting.
See below. I've done a little bit of cleanup, but this is what you could do...
I'm reading the contents of the file into memory once, storing it as a variable. Then performing the replacements on the variable, and then writing the contents of the variable to file:
$inputTextFilePath = 'D:\StackOverflow\fart\test.txt'
$outputTextFilePath = 'D:\StackOverflow\fart\test_output.txt'
class FindReplace {
[string]$FindString;
[string]$ReplaceString;
}
[System.Collections.Generic.List[FindReplace]]$FindReplaceList = #()
$Obj1 = New-Object FindReplace
$Obj1.FindString = "AVERAGE"
$Obj1.ReplaceString = "AVG"
$Obj2 = New-Object FindReplace
$Obj2.FindString = "TEMPERATURE"
$Obj2.ReplaceString = "TEMP"
$FindReplaceList.Add($Obj1);
$FindReplaceList.Add($Obj2);
write-host $FindReplaceList.Count
$fileContent = Get-Content $inputTextFilePath;
foreach ($item in $FindReplaceList)
{
Write-Host "Replacing string $($item.FindString) to $($item.ReplaceString) where input file: $inputTextFilePath and output file: $outputTextFilePath";
$fileContent = $fileContent.Replace($item.FindString, $item.ReplaceString);
}
Set-Content -Value $fileContent -Path $outputTextFilePath;
And of course, since "[F]ind [A]nd [R]eplace [T]ext" questions only come up every so often...I had to make use of that fancy acronym :)

Related

Adding a column to a datatable in powershell

I am trying to add a column to data I have imported (and will export) as a CSV.
I am importing a CSV:
What I want to do add another column, perhaps "10/15/22" when the process runs, and then update the values under that date.
In effect, the document will grow to the right, adding a column each time it is run.
I have an object called $test. It will have values like:
$test.users = "7"
$test.SomeBSValue = "22"
$test.Computers = "52"
When all is said and done, my output should look like:
Adding to the list any values I have that are not part of the list already, but recording the values I have under the heading for the date.
So, if the script is run and collects 100 data points, those data point would all be in the CSV under the date.
I would have thought this would be easy, but now I am drawing a complete blank.
I've considered (but have not coded) even trying to put into a GUI grid view and then reading the data back and writing the CSV (but there should be an easier way, right?)
Since you don't actually use it as a CSV we can treat it like regular content.
Say we have a file in C:\test called test.csv that looks as follows:
"Settings","08/15/22","09/15/22"
"Users",0,0
"Computers",0,1
"SomeValue1",0,2
"SomeValue2",0,2
"SomeValue3",0,2
"Stat1",0,10
"Stat2",7,0
"Stat3",0,0
"SomeBSValue",1,2
We can import it, add the row from the object to each corresponding row and right the file to test2.csv.
$test = #{
Settings = "10/15/22"
users = "7"
Computers = "52"
SomeValue1 = "22"
SomeValue2 = "24"
SomeValue3 = "25"
Stat1 = "4"
Stat2 = "3"
Stat3 = "2"
SomeBSValue = "1"
}
$content = Get-Content "C:\test\test.csv"
$newContent = #()
foreach($row in $content){
foreach($key in $test.Keys){
if($row -like "*$key*"){
$row = $row + "," + $test."$key"
$newContent += $row
}
}
}
$newContent | Out-File "C:\test\test2.csv"
After running the script it will have added the values from the object:
"Settings","08/15/22","09/15/22",10/15/22
"Users",0,0,7
"Computers",0,1,52
"SomeValue1",0,2,22
"SomeValue2",0,2,22
"SomeValue3",0,2,22
"Stat1",0,10,4
"Stat2",7,0,4
"Stat3",0,0,4
Edit:
If you want the date between quotes, replace $row = $row + "," + $test."$key" with this:
if($key -eq "Settings"){
$row = $row + "," + '"' + $test."$key" + '"'
}else{
$row = $row + "," + $test."$key"
}
This idea is pretty terrible idea, as you stated, "grow to the right" is definitely not a good approach and you should consider a better way of doing it, data should always expand vertically.
As for the solution, you can create new columns easily with Select-Object and dynamically generated calculated properties.
Note, this should definitely not be considered an efficient approach. This will be slow because Select-Object is slow.
function Add-Column {
param(
[Parameter(ValueFromPipeline, DontShow, Mandatory)]
[object] $InputObject,
[Parameter(Mandatory)]
[string] $ColumnName,
[Parameter(Mandatory)]
[string] $ReferenceProperty,
[Parameter(Mandatory)]
[hashtable] $Values
)
begin {
$calculatedProp = #{ N = $ColumnName }
}
process {
$calculatedProp['E'] = { 0 }
if($value = $InputObject.$ReferenceProperty) {
if($Values.ContainsKey($value)) {
$calculatedProp['E'] = { $Values[$value] }
}
}
$InputObject | Select-Object *, $calculatedProp
}
}
Usage
Import-Csv path\to\csv | Add-Column -ColumnName '09/15/22' -ReferenceProperty Settings -Values #{
users = "7"
SomeBSValue = "22"
Computers = "52"
}
Result
Settings 08/15/22 09/15/22
-------- -------- --------
Users 0 7
Computers 0 52
SomeValue1 0 0
SomeValue2 0 0
SomeValue3 0 0
Stat1 0 0
Stat2 7 0
Stat3 0 0
SomeBSValue 1 22
This function allows then pipe into Export-Csv at ease:
Import-Csv path\to\csv | Add-Column ... | Export-Csv path\to\newCsv

Windows PowerShell: How to parse the log file?

I have an input file with below contents:
27/08/2020 02:47:37.365 (-0516) hostname12 ult_licesrv ULT 5 LiceSrv Main[108 00000 Session 'session1' (from 'vmpms1\app1#pmc21app20.pm.com') request for 1 additional licenses for module 'SA-XT' - 1 licenses have been allocated by concurrent usage category 'Unlimited' (session module usage now 1, session category usage now 1, total module concurrent usage now 1, total category usage now 1)
27/08/2020 02:47:37.600 (-0516) hostname13 ult_licesrv ULT 5 LiceSrv Main[108 00000 Session 'sssion2' (from 'vmpms2\app1#pmc21app20.pm.com') request for 1 additional licenses for module 'SA-XT-Read' - 1 licenses have been allocated by concurrent usage category 'Floating' (session module usage now 2, session category usage now 2, total module concurrent usage now 1, total category usage now 1)
27/08/2020 02:47:37.115 (-0516) hostname141 ult_licesrv CMN 5 Logging Housekee 00000 Deleting old log file 'C:\Program Files\PMCOM Global\License Server\diag_ult_licesrv_20200824_011130.log.gz' as it exceeds the purge threashold of 72 hours
27/08/2020 02:47:37.115 (-0516) hostname141 ult_licesrv CMN 5 Logging Housekee 00000 Deleting old log file 'C:\Program Files\PMCOM Global\License Server\diag_ult_licesrv_20200824_021310.log.gz' as it exceeds the purge threashold of 72 hours
27/08/2020 02:47:37.625 (-0516) hostname150 ult_licesrv ULT 5 LiceSrv Main[108 00000 Session 'session1' (from 'vmpms1\app1#pmc21app20.pm.com') request for 1 additional licenses for module 'SA-XT' - 1 licenses have been allocated by concurrent usage category 'Unlimited' (session module usage now 2, session category usage now 1, total module concurrent usage now 2, total category usage now 1)
I need to generate and output file like below:
Date,time,hostname,session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage
27/08/2020,02:47:37.365 (-0516),hostname12,1,1,1,1
27/08/2020,02:47:37.600 (-0516),hostname13,2,2,1,1
27/08/2020,02:47:37.115 (-0516),hostname141,0,0,0,0
27/08/2020,02:47:37.115 (-0516),hostname141,0,0,0,0
27/08/2020,02:47:37.625 (-0516),hostname150,2,1,2,1
The output data order is: Date,time,hostname,session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage.
Put 0,0,0,0 if no entry for session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage
I need to get content from the input file and write the output to another file.
Update
I have created a file input.txt in F drive and pasted the log details into it.
Then I form an array by splitting the file content when a new line occurs like below.
$myList = (Get-Content -Path F:\input.txt) -split '\n'
Now I got 5 items in my array myList. Then I replace the multiple blank spaces with a single blank space and formed a new array by splitting each element by blank space. Then I print the 0 to 3 array elements. Now I need to add the end values (session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage).
PS C:\Users\user> $myList = (Get-Content -Path F:\input.txt) -split '\n'
PS C:\Users\user> $myList.Length
5
PS C:\Users\user> $myList = (Get-Content -Path F:\input.txt) -split '\n'
PS C:\Users\user> $myList.Length
5
PS C:\Users\user> for ($i = 0; $i -le ($myList.length - 1); $i += 1) {
>> $newList = ($myList[$i] -replace '\s+', ' ') -split ' '
>> $newList[0]+','+$newList[1]+' '+$newList[2]+','+$newList[3]
>> }
27/08/2020,02:47:37.365 (-0516),hostname12
27/08/2020,02:47:37.600 (-0516),hostname13
27/08/2020,02:47:37.115 (-0516),hostname141
27/08/2020,02:47:37.115 (-0516),hostname141
27/08/2020,02:47:37.625 (-0516),hostname150
If you really need to filter on the granularity that you're looking for, then you may need to use regex to filter the lines.
This would assume that the rows have similarly labeled lines before the values you're looking for, so keep that in mind.
[System.Collections.ArrayList]$filteredRows = #()
$log = Get-Content -Path C:\logfile.log
foreach ($row in $log) {
$rowIndex = $log.IndexOf($row)
$date = ([regex]::Match($log[$rowIndex],'^\d+\/\d+\/\d+')).value
$time = ([regex]::Match($log[$rowIndex],'\d+:\d+:\d+\.\d+\s\(\S+\)')).value
$hostname = ([regex]::Match($log[$rowIndex],'(?<=\d\d\d\d\) )\w+')).value
$sessionModuleUsage = ([regex]::Match($log[$rowIndex],'(?<=session module usage now )\d')).value
if (!$sessionModuleUsage) {
$sessionModuleUsage = 0
}
$sessionCategoryUsage = ([regex]::Match($log[$rowIndex],'(?<=session category usage now )\d')).value
if (!$sessionCategoryUsage) {
$sessionCategoryUsage = 0
}
$moduleConcurrentUsage = ([regex]::Match($log[$rowIndex],'(?<=total module concurrent usage now )\d')).value
if (!$moduleConcurrentUsage) {
$moduleConcurrentUsage = 0
}
$totalCategoryUsage = ([regex]::Match($log[$rowIndex],'(?<=total category usage now )\d')).value
if (!$totalCategoryUsage) {
$totalCategoryUsage = 0
}
$hash = [ordered]#{
Date = $date
time = $time
hostname = $hostname
session_module_usage = $sessionModuleUsage
session_category_usage = $sessionCategoryUsage
module_concurrent_usage = $moduleConcurrentUsage
total_category_usage = $totalCategoryUsage
}
$rowData = New-Object -TypeName 'psobject' -Property $hash
$filteredRows.Add($rowData) > $null
}
$csv = $filteredRows | convertto-csv -NoTypeInformation -Delimiter "," | foreach {$_ -replace '"',''}
$csv | Out-File C:\results.csv
What essentially needs to happen is that we need to get-content of the log, which returns an array with each item terminated on a newline.
Once we have the rows, we need to grab the values via regex
Since you want zeroes in some of the items if those values don't exist, I have if statements that assign '0' if the regex returns nothing
Finally, we add each filtered item to a PSObject and append that object to an array of objects in each iteration.
Then export to a CSV.
You can probably pick apart the lines with a regex and substrings easily enough. Basically something like the following:
# Iterate over the lines of the input file
Get-Content F:\input.txt |
ForEach-Object {
# Extract the individual fields
$Date = $_.Substring(0, 10)
$Time = $_.Substring(12, $_.IndexOf(')') - 11)
$Hostname = $_.Substring(34, $_.IndexOf(' ', 34) - 34)
$session_module_usage = 0
$session_category_usage = 0
$module_concurrent_usage = 0
$total_category_usage = 0
if ($_ -match 'session module usage now (\d+), session category usage now (\d+), total module concurrent usage now (\d+), total category usage now (\d+)') {
$session_module_usage = $Matches[1]
$session_category_usage = $Matches[2]
$module_concurrent_usage = $Matches[3]
$total_category_usage = $Matches[4]
}
# Create custom object with those properties
New-Object PSObject -Property #{
Date = $Date
time = $Time
hostname = $Hostname
session_module_usage = $session_module_usage
session_category_usage = $session_category_usage
module_concurrent_usage = $module_concurrent_usage
total_category_usage = $total_category_usage
}
} |
# Ensure column order in output
Select-Object Date,time,hostname,session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage |
# Write as CSV - without quotes
ConvertTo-Csv -NoTypeInformation |
ForEach-Object { $_ -replace '"' } |
Out-File F:\output.csv
Whether to pull the date, time, and host name from the line with substrings or regex is probably a matter of taste. Same goes for how strict the format must be matched, but that to me mostly depends on how rigid the format is. For more free-form things where different lines would match different regexes, or multiple lines makes up a single record, I also quite like switch -Regex to iterate over the lines.

is there a simple way to output to xlsx?

I am trying to output a query from a DB to a xlsx but it takes so much time to do this because there about 20,000 records to process, is there a simpler way to do this?
I know there is a way to do it for csv but im trying to avoid that, because if the records had any comma is going to take it as a another column and that would mess with the info
this is my code
$xlsObj = New-Object -ComObject Excel.Application
$xlsObj.DisplayAlerts = $false
$xlsWb = $xlsobj.Workbooks.Add(1)
$xlsObj.Visible = 0 #(visible = 1 / 0 no visible)
$xlsSh = $xlsWb.Worksheets.Add([System.Reflection.Missing]::Value, $xlsWb.Worksheets.Item($xlsWb.Worksheets.Count))
$xlsSh.Name = "QueryResults"
$DataSetTable= $ds.Tables[0]
Write-Output "DATA SET TABLE" $DataSetTable
[Array] $getColumnNames = $DataSetTable.Columns | SELECT *
Write-Output "COLUMN NAMES" $DataSetTable.Rows[0]
[Int] $RowHeader = 1
foreach ($ColH in $getColumnNames)
{
$xlsSh.Cells.item(1, $RowHeader).font.bold = $true
$xlsSh.Cells.item(1, $RowHeader) = $ColH.ColumnName
Write-Output "Nombre de Columna"$ColH.ColumnName
$RowHeader++
}
[Int] $rowData = 2
[Int] $colData = 1
foreach ($rec in $DataSetTable.Rows)
{
foreach ($Coln in $getColumnNames)
{
$xlsSh.Cells.NumberFormat = "#"
$xlsSh.Cells.Item($rowData, $colData) = $rec.$($Coln.ColumnName).ToString()
$ColData++
}
$rowData++; $ColData = 1
}
$xlsRng = $xlsSH.usedRange
[void] $xlsRng.EntireColumn.AutoFit()
#Se elimina la pestaña Sheet1/Hoja1.
$xlsWb.Sheets(1).Delete() #Versión 02
$xlsFile = "directory of the file"
[void] $xlsObj.ActiveWorkbook.SaveAs($xlsFile)
$xlsObj.Quit()
Start-Sleep -Milliseconds 700
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($xlsRng)) {''}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($xlsSh)) {''}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($xlsWb)) {''}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($xlsObj)) {''}
[gc]::collect() | Out-Null
[gc]::WaitForPendingFinalizers() | Out-Null
$oraConn.Close()
I'm trying to avoid [CSV files], because if the records had any comma is going to take it as a another column and that would mess with the info
That's only the case if you try to construct the output format manually. Builtin commands like Export-Csv and ConvertTo-Json will automatically quote the values as necessary:
PS C:\> $customObject = [pscustomobject]#{ID = 1; Name = "Solis, Heber"}
PS C:\> $customObject
ID Name
-- ----
1 Solis, Heber
PS C:\> $customObject |ConvertTo-Csv -NoTypeInformation
"ID","Name"
"1","Solis, Heber"
Notice, in the example above, how:
The string value assigned to $customObject.Name does not contain any quotation marks, but
In the output from ConvertTo-Csv we see values and headers clearly enclosed in quotation marks
PowerShell automatically enumerates the row data when you pipe a [DataTable] instance, so creating a CSV might (depending on the contents) be as simple as:
$ds.Tables[0] |Export-Csv table_out.csv -NoTypeInformation
What if you want TAB-separated values (or any other non-comma separator)?
The *-Csv commands come with a -Delimiter parameter to which you can pass a user-defined separator:
# This produces semicolon-separated values
$data |Export-Csv -Path output.csv -Delimiter ';'
I usually try and refrain from recommending specific modules libraries, but if you insist on writing to XSLX I'd suggest checking out ImportExcel (don't let the name fool you, it does more than import from excel, including exporting and formatting data from PowerShell -> XSLX)

comparision of 2 csv files using powershell

The format of two files is same and as follows:
ServiceName Status computer State
AdobeARMservice OK NEE Running
Amazon Assistan OK NEE Running
the requirement is, i have to check the service name and computer name..if both are same, then i have to check whether the state of particular service is same in both the files or not. And if it is not same then display it..
$preser = import-csv C:\info.csv
$postser = import-csv C:\serviceinfo.csv
foreach($ser1 in $preser)
{
foreach($ser2 in $postser)
{
if(($ser1.computer -eq $ser2.computer) -and ($ser1.ServiceName -eq $ser2.ServiceName))
{
if($ser1.State -eq $ser2.State)
{
}
else
{
write-host $ser1,$ser2
}
}
}
}
This code is working fine but as the files length is very large, the time of execution is more.
Is there any alternative method to reduce the time of execution..?
Thank you
Although Import-Csv on very large files will take its time, maybe this will be faster:
$preser = Import-Csv -Path 'C:\info.csv'
$postser = Import-Csv -Path 'C:\serviceinfo.csv'
# build a lookup Hashtable for $preser
$hash = #{}
foreach ($item in $preser) {
# combine the ServiceName and Computer to form the hash key
$key = '{0}#{1}' -f $item.ServiceName, $item.computer
$hash[$key] = $item
}
# now loop through the items in $postser
foreach ($item in $postser) {
$key = '{0}#{1}' -f $item.ServiceName, $item.computer
if ($hash.ContainsKey($key)) {
if ($hash[$key].State -ne $item.State) {
# create a new object for output
$out = $hash[$key] | Select-Object * -ExcludeProperty State
$out | Add-Member -MemberType NoteProperty -Name 'State in Preser' -Value $hash[$key].State
$out | Add-Member -MemberType NoteProperty -Name 'State in Postser' -Value $item.State
$out
}
}
}
The output on screen will look something like this:
ServiceName : AdobeARMservice
Status : OK
computer : NEE
State in Preser : Running
State in Postser : Stopped
Of course, you can capture this output and save it as new csv if you do
$result = foreach ($item in $postser) {
# rest of the above foreach loop
}
# output on screen
$result
# output to new csv
$result | Export-Csv -Path 'C:\ServiceInfoDifference.csv' -NoTypeInformation
There are a few ways to do this:
1. Sorting the columns
If the columns are unsorted in the files, sort them first, and then try finding a match by using linear search.
2. Binary search
What you are currently doing is an implementation of a linear search. You can implement binary search (works best on sorted lists) to find a result faster.
Taken from dfinkey's github repo
function binarySearch {
param($sortedArray, $seekElement, $comparatorCallback)
$comparator = New-Object Comparator $comparatorCallback
$startIndex = 0
$endIndex = $sortedArray.length - 1
while ($startIndex -le $endIndex) {
$middleIndex = $startIndex + [Math]::floor(($endIndex - $startIndex) / 2)
# If we've found the element just return its position.
if ($comparator.equal($sortedArray[$middleIndex], $seekElement)) {
return $middleIndex
}
# Decide which half to choose for seeking next: left or right one.
if ($comparator.lessThan($sortedArray[$middleIndex], $seekElement)) {
# Go to the right half of the array.
$startIndex = $middleIndex + 1
}
else {
# Go to the left half of the array.
$endIndex = $middleIndex - 1
}
}
return -1
}
3. Hashes
I am not completely sure of this method, but, you can load the columns into hashes and then compare them. Hash comparisons are generally faster than array comparisons.

PowerShell unable to get file metadata from Comments field when it's too long

I want to extract some xml data from the Comments metadata field in .WMA files.
I'm using a script from Technet's Scripting Guy column to get all metadata, and it lists every attribute except the Comments field!
Some research by my colleague showed that when we shortened the data in the Comments field to < 1024 bytes, the data from the Comments field lists out fine.
It seems to me that the limitation is in the Shell.Application object; it just returns an empty Comments field when the contents is more than 1024 characters. Also, instead of listing every attribute, I just get the Comments, which is number 24.
The sample file I have contains 1188 bytes, and I think files will be aruond there, so it's not over by much.
Here is the script I'm currently running (removed comments for brevity):
Function Get-FileMetaData
{
Param([string[]]$folder)
foreach($sFolder in $folder)
{
$a = 0
$objShell = New-Object -ComObject Shell.Application
$objFolder = $objShell.namespace($sFolder)
foreach ($File in $objFolder.items())
{
$FileMetaData = New-Object PSOBJECT
$hash += #{"Filename" = $($objFolder.getDetailsOf($File, 0)) }
$hash += #{"My Comment field" = $($objFolder.getDetailsOf($File, 24)) }
$hash += #{"Length" = $($objFolder.getDetailsOf($File, 24)).Length }
$FileMetaData | Add-Member $hash
$hash.clear()
} #end foreach
$a=0
$FileMetaData
} #end foreach $sfolder
}
Get-FileMetaData -folder "C:\DATA\wma" | fl
Is there another approach I can use that will allow me to extract the full XML data?
you can try to use the taglib-sharp dll from http://taglib.org/
here I copy the content of a 156 KB file to the comment :
[system.reflection.assembly]::loadfile("c:\temp\taglib-sharp.dll")
$data=[taglib.file]::create('c:\mp3\01. Stromae - Alors On Danse.mp3')
$data.Tag.Comment = (gc c:\temp\IMP_ERR.LOG)
$data.Save()
verification :
PS>$data=[taglib.file]::create('c:\mp3\01. Stromae - Alors On
Danse.mp3') PS>$data.tag.Comment.length / 1KB
PS>155,2197265625
edit
I was able to use same code for a wma file