I have small error when running my code. I assign a string to custom object but it's parsing the string by itself and throwing an error.
Code:
foreach ($item in $hrdblistofobjects) {
[string]$content = Get-Content -Path $item
[string]$content = $content.Replace("[", "").Replace("]", "")
#here is line 43 which is shown as error as well
foreach ($object in $listofitemsdb) {
$result = $content -match $object
$OurObject = [PSCustomObject]#{
ObjectName = $null
TestObjectName = $null
Result = $null
}
$OurObject.ObjectName = $item
$OurObject.TestObjectName = $object #here is line 52 which is other part of error
$OurObject.Result = $result
$Resultsdb += $OurObject
}
}
This code loads an item and checks if an object exists within an item. Basically if string part exists within a string part and then saves result to a variable. I am using this code for other objects and items but they don't have that \p part which I am assuming is the issue. I can't put $object into single quotes for obvious reasons (this was suggested on internet but in my case it's not possible). So is there any other option how to unescape \p? I tried $object.Replace("\PMS","\\PMS") but that did not work either (this was suggested somewhere too).
EDIT:
$Resultsdb = #(foreach ($item in $hrdblistofobjects) {
[string]$content = Get-Content -Path $item
[string]$content = $content.Replace("[", "").Replace("]", "")
foreach ($object in $listofitemsdb) {
[PSCustomObject]#{
ObjectName = $item
TestObjectName = $object
Result = $content -match $object
}
}
}
)
$Resultsdb is not defined as an array, hence you get that error when you try to add one object to another object when that doesn't implement the addition operator.
You shouldn't be appending to an array in a loop anyway. That will perform poorly, because with each iteration it creates a new array with the size increased by one, copies all elements from the existing array, puts the new item in the new free slot, and then replaces the original array with the new one.
A better approach is to just output your objects in the loop and collect the loop output in a variable:
$Resultsdb = foreach ($item in $hrdblistofobjects) {
...
foreach ($object in $listofitemsdb) {
[PSCustomObject]#{
ObjectName = $item
TestObjectName = $object
Result = $content -match $object
}
}
}
Run the loop in an array subexpression if you need to ensure that the result is an array, otherwise it will be empty or a single object when the loop returns less than two results.
$Resultsdb = #(foreach ($item in $hrdblistofobjects) {
...
})
Note that you need to suppress other output on the default output stream in the loop, so that it doesn't pollute your result.
I changed the match part to this and it's working fine $result = $content -match $object.Replace("\PMS","\\PMS").
Sorry for errors in posting. I will amend that.
Related
I have a TXT file with 1300 megabytes (huge thing). I want to build code that does two things:
Every line contains a unique ID at the beginning. I want to check for all lines with the same unique ID if the conditions is met for that "group" of IDs. (This answers me: For how many lines with the unique ID X have all conditions been met)
If the script is finished I want to remove all lines from the TXT where the condition was met (see 2). So I can rerun the script with another condition set to "narrow down" the whole document.
After few cycles I finally have a set of conditions that applies to all lines in the document.
It seems that my current approach is very slow.( one cycle needs hours). My final result is a set of conditions that apply to all lines of code.
If you find an easier way to do that, feel free to recommend.
Help is welcome :)
Code so far (does not fullfill everything from 1&2)
foreach ($item in $liste)
{
# Check Conditions
if ( ($item -like "*XXX*") -and ($item -like "*YYY*") -and ($item -notlike "*ZZZ*")) {
# Add a line to a document to see which lines match condition
Add-Content "C:\Desktop\it_seems_to_match.txt" "$item"
# Retrieve the unique ID from the line and feed array.
$array += $item.Split("/")[1]
# Remove the line from final document
$liste = $liste -replace $item, ""
}
}
# Pipe the "new cleaned" list somewhere
$liste | Set-Content -Path "C:\NewListToWorkWith.txt"
# Show me the counts
$array | group | % { $h = #{} } { $h[$_.Name] = $_.Count } { $h } | Out-File "C:\Desktop\count.txt"
Demo Lines:
images/STRINGA/2XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGA/3XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGB/4XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGB/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGC/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
performance considerations:
Add-Content "C:\Desktop\it_seems_to_match.txt" "$item"
try to avoid wrapping cmdlet pipelines
See also: Mastering the (steppable) pipeline
$array += $item.Split("/")[1]
Try to avoid using the increase assignment operator (+=) to create a collection
See also: Why should I avoid using the increase assignment operator (+=) to create a collection
$liste = $liste -replace $item, ""
This is a very expensive operation considering that you are reassigning (copying) a long list ($liste) with each iteration.
Besides it is a bad practice to change an array that you are currently iterating.
$array | group | ...
Group-Object is a rather slow cmdlet, you better collect (or count) the items on-the-fly (where you do $array += $item.Split("/")[1]) using a hashtable, something like:
$Name = $item.Split("/")[1]
if (!$HashTable.Contains($Name)) { $HashTable[$Name] = [Collections.Generic.List[String]]::new() }
$HashTable[$Name].Add($Item)
To minimize memory usage it may be better to read one line at a time and check if it already exists. Below code I used StringReader and you can replace with StreamReader for reading from a file. I'm checking if the entire string exists, but you may want to split the line. Notice I have duplicaes in the input but not in the dictionary. See code below :
$rows= #"
images/STRINGA/2XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGA/3XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/4XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGC/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGA/2XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGA/3XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/4XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGC/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
"#
$dict = [System.Collections.Generic.Dictionary[int, System.Collections.Generic.List[string]]]::new();
$reader = [System.IO.StringReader]::new($rows)
while(($row = $reader.ReadLine()) -ne $null)
{
$hash = $row.GetHashCode()
if($dict.ContainsKey($hash))
{
#check if list contains the string
if($dict[$hash].Contains($row))
{
#string is a duplicate
}
else
{
#add string to dictionary value if it is not in list
$list = $dict[$hash].Value
$list.Add($row)
}
}
else
{
#add new hash value to dictionary
$list = [System.Collections.Generic.List[string]]::new();
$list.Add($row)
$dict.Add($hash, $list)
}
}
$dict
I'm trying add to a variable and a string in an array dynamically but i'm not getting expected output.
(1) I'm getting env name
(2) Concatinating the string and variable in an array
Code is as follows.
$env = $env:COMPUTERNAME.Substring(0,2)
$servers = { $env+"server1.test.com",$env+"server2.test.com" }
$serverCount = $servers -split(",") | measure | % { $_.Count }
For ($i=0; $i -lt $serverCount; $i++)
{
$ServerName = $servers -split(',') -replace '\[\d+\]'
$server = $ServerName[$i]
Write-Host $server
}
output i'm getting as
$env+"server1.test.com"
$env+"server2.test.com"
Values are not getting concatenated properly and variable value is not getting displayed. Any help.
$servers = { $env+"server1.test.com",$env+"server2.test.com" }
This is a scriptblock, not an array. {} is like a function, you have to run it for it to do anything (such as evaluating $env).
When you force it into a string using -split(",") what you get is text representation of the source code in the scriptblock, including the variable names.
As #Olaf comments, the right way to create an array of names is
$servers = ($env + "server1.test.com"), ($env + "server2.test.com")
This might be how I'd write it:
$env = $env:COMPUTERNAME.Substring(0,2)
"server1.test.com", "server2.test.com" | foreach-object {
"$env$_" -replace '\d+'
}
I have a large CSV file in which some fields have a new line embedded. Excel 2016 produces errors when importing a CSV with rows which have fields with a new line embedded.
Based on this post, I wrote code to replace any new line in any field with a space. Below is a code block that duplicates the functionality and issue. Option 1 works. Option 2, which is commented out, casts my object to a string. I was hoping Option 2 might run faster.
Question: Is there a better way to do this to optimize for performance processing very large files?
$array = #([PSCustomObject]#{"ID"="1"; "Name"="Joe`nSmith"},
[PSCustomObject]#{"ID"="2"; "Name"="Jasmine Baker"})
$array = $array | ForEach-Object {
#Option 1: produces an Object, but is code optimized?
foreach ($n in $_.PSObject.Properties.Name) {
$_.PSObject.Properties[$n].Value = `
$_.PSObject.Properties[$n].Value -replace "`n"," "
}
#Option 2: produces a string, not an object
#$_ = $_ -replace "`n"," "
$_
}
Keep in mind that in my real-world use case, each row has > 15 fields and any combination of them may have one or more new lines embedded.
Use the fast TextFieldParser to read, process, and build the CSV from the file (PowerShell 3+):
[Reflection.Assembly]::LoadWithPartialName('Microsoft.VisualBasic') >$null
$parser = New-Object Microsoft.VisualBasic.FileIO.TextFieldParser 'r:\1.csv'
$parser.SetDelimiters(',')
$header = $parser.ReadFields()
$CSV = while (!$parser.EndOfData) {
$i = 0
$row = [ordered]#{}
foreach ($field in $parser.ReadFields()) {
$row[$header[$i++]] = $field.replace("`n", ' ')
}
[PSCustomObject]$row
}
Or modify each field in-place in an already existing CSV array:
foreach ($row in $CSV) {
foreach ($field in $row.PSObject.Properties) {
$field.value = $field.value.replace("`n", ' ')
}
}
Notes:
foreach statement is much faster than piping to ForEach-Object (also aliased as foreach)
$stringVariable.replace() is faster then -replace operator
Say I have 2 powershell hashtables one big and one small and, for a specific purpose I want to say they are equal if for the keys in the small one, the keys on the big hastable are the same.
Also I don't know the names of the keys in advance. I can use the following function that uses Invoke-Expression but I am looking for nicer solutions, that don't rely on this.
Function Compare-Subset {
Param(
[hashtable] $big,
[hashtable] $small
)
$keys = $small.keys
Foreach($k in $keys) {
$expression = '$val = $big.' + "$k" + ' -eq ' + '$small.' + "$k"
Invoke-Expression $expression
If(-not $val) {return $False}
}
return $True
}
$big = #{name='Jon'; car='Honda'; age='30'}
$small = #{name = 'Jon'; car='Honda'}
Compare-Subset $big $small
A simple $true/$false can easily be gotten. This will return $true if there are no differences:
[string]::IsNullOrWhiteSpace($($small|Select -Expand Keys|Where{$Small[$_] -ne $big[$_]}))
It checks for all keys in $small to see if the value of that key in $small is the same of the value for that key in $big. It will only output any values that are different. It's wrapped in a IsNullOrWhitespace() method from the [String] type, so if any differences are found it returns false. If you want to list differences just remove that method.
This could be the start of something. Not sure what output you are looking for but this will output the differences between the two groups. Using the same sample data that you provided:
$results = Compare-Object ($big.GetEnumerator() | % { $_.Name }) ($small.GetEnumerator() | % { $_.Name })
$results | ForEach-Object{
$key = $_.InputObject
Switch($_.SideIndicator){
"<="{"Only reference object has the key: '$key'"}
"=>"{"Only difference object has the key: '$key'"}
}
}
In primetime you would want something different but just to show you the above would yield the following output:
Only reference object has the key: 'age'
What I'm trying to do is create array variable names dynamically, and then with a loop, add the object to its relevant array based on the hash table value being equal to the counter variable.
$hshSite = #{} # Values like this CO,1 NE,2 IA,3
$counter = $hshSite.count
For($i = $counter; $i -gt 0; $i--) {
New-Variable -Name "arr$i" -Value #()
}
If $counter = 3, I would create arrays $arr1, $arr2, $arr3
$csv = Import-CSV....
ForEach ($x in $csv) {
#if $hshSite.Name = $x.location (ie CO), look up hash value (1),
and add the object to $arr1. If $hshSite.Name = NE, add to $arr2
I tried creating the dynamic arrays with New-Variable, but having issues trying to add to those arrays. Is it possible to concatenate 2 variables names into a single variable name? So taking $arr + $i to form $arr1 and $arr2 and $arr3, and then I can essentially just do $arr0 += $_
The end goal is to group things based on CO, NE, IA for further sorting/grouping/processing. And I'm open to other ideas of getting this accomplished. Thanks for your help!
Just make your hash table values the arrays, and accumulate the values to them directly:
$Sites = 'CO','NE','IA'
$hshSite = #{}
Foreach ($Site in $Sites){$hshSite[$Site] = #()}
ForEach ($x in $csv)
{
$hshSite[$x.location] += <whatever it is your adding>
}
If there's a lot of entries in the csv, you might consider creating those values as arraylists instead of arrays.
$Sites = 'CO','NE','IA'
$hshSite = #{}
Foreach ($Site in $Sites){ $hshSite[$Site] = New-Object Collections.Arraylist }
ForEach ($x in $csv)
{
$hshSite[$x.location].add('<whatever it is your adding>') > $nul
}
You could quite easily do add items to a dynamically named array variable using the Get-Variable cmdlet. Similar to the following:
$MyArrayVariable123 = #()
$VariableNamePrefix = "MyArrayVariable"
$VariableNameNumber = "123"
$DynamicallyRetrievedVariable = Get-Variable -Name ($VariableNamePrefix + $VariableNameNumber)
$DynamicallyRetrievedVariable.Value += "added item"
After running the above code the $MyArrayVariable123 variable would be an array holding the single string added item.