I'm working with a hashtable which I've built using a list of 3.5 million IP addresses stored in CSV format, and I am trying to search through this table using wildcards.
The CSV is MaxMind's list of IPs, which I convert to Hashtable using the following code
[System.IO.File]::ReadLines("C:\temp\iptest.csv") | ForEach-Object { $data= $_.split(','); $ht = #{"geoname_id"="$($data[1])";"registered_country_geoname_id"="$($data[2])"}
$name = $($data[0])
$mainIPHhash.add($name, $ht)}
The code just pulls out the CIDR and it's corresponding City/Country code.
This works well, and builds the table in a little over two minutes, but the issue I am now facing is searching this hashtable for wild card entries.
If I search for a complete CIDR, the search happens in milliseconds
$mainIPHhash.item("1.0.0.0/24")
Measure command reports - TotalSeconds : 0.0001542
But if I need to do a wildcard search, it has to loop through the hashtable looking for my like values, which takes a long time!
$testingIP = "1.0.*"
$mainIPHhash.GetEnumerator() | Where-Object { $_.key -like $testingIP }
Measure command reports - TotalSeconds : 33.3016279
Is there a better way for searching wildcard entries in Hashtables?
Cheers
Edit:
Using a regex search, I can get it down to 19 seconds. But still woefully slow
$findsStr = "^$(($testingIP2).split('.')[0])" +"\."+ "$(($testingIP2).split('.')[1])" +"\."
$mainIPHhash.GetEnumerator() | foreach {if($_.Key -match $findsStr){#Dostuff }}
The above takes the first two octets of the IP address, and uses regex to find them in the hashtable.
Days : 0
Hours : 0
Minutes : 0
Seconds : 19
Milliseconds : 733
Ticks : 197339339
TotalDays : 0.000228402012731481
TotalHours : 0.00548164830555556
TotalMinutes : 0.328898898333333
TotalSeconds : 19.7339339
TotalMilliseconds : 19733.9339
You can take the list of IPs and do either -like or -match for a list. Either should be faster than a Where-Object clause
$mainIPhash.Values -like '1.0.*'
$mainIPhash.Values -match '^1\.0\.'
Other solution may be, use group-object :
$contentcsv=import-csv "C:\temp\iptest.csv" -Header Name, geoname_id, registered_country_geoname_id |Group Name
$contentcsv | where Name -like '1.0.*'
I just wanted a .ps1 file that will run a simple line of powershell but not close instantly.
Ive tried to do "read-host -prompt " " " but it is displaying before the code is run and then still closes instantly
get-appxpackage -allusers | select name
read-host -prompt "Press enter to exit"
I expect the outcome to be I run the file and then get a chance to read the output within the powershell window before pressing something to exit. But the actual output is prompts to exit before the code is run and then it runs through the output and closes
After executing this line of code:
get-appxpackage -allusers | select name
You'll have some "pending" objects ready to return to the Powershell pipelines output stream. The objects can't be sent to the pipeline until Read-Host has finished (since Powershell will treat these objects as "output" of your ps1 file). After Read-Host has finished the objects are sent to the pipeline (via the output stream). Since there is no other cmdlet there (using the output of you ps1 file), Powershells default behavior is to output the pipeline content to the Powershell host.
As #Lee_Daily already mentioned in above comment, adding Out-Host will send the output of get-appxpackage -allusers | select name to the Powershell host. So get-appxpackage -allusers | select name | out-host no objects are queued in the output stream for further pipeline actions.
I would recommend you check following sources:
About pipeline
About redirection
Understanding streams
These are essential Powershell concepts you've to understand.
Hope that helps.
Tagging on to what the Lee and Moerwald have said.
Another way to stream real-time is using ForEach or ForEach-Object, it's also a bit more performant than the Out-Host approach, because of the not defaulting to writing to the screen. If that latter performance v screen write is important to you. If you don't want the screen output of Out-Host, send it to Null.
# Using Out-Host
Measure-Command {
get-appxpackage -allusers |
select name | Out-Host
}
Name
----
1527c705-...
...
InputApp
Microsoft.AAD.BrokerPlugin
Microsoft.AccountsControl
...
# Results
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
*** Milliseconds : 643 ***
Ticks : 6431627
TotalDays : 7.44401273148148E-06
TotalHours : 0.000178656305555556
TotalMinutes : 0.0107193783333333
TotalSeconds : 0.6431627
TotalMilliseconds : 643.1627
# User ForEach in a script Block
Measure-Command {
& { foreach ($item in get-appxpackage -allusers | select name)
{ "processing $item"}}
}
# Results
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
*** Milliseconds : 385 ***
Ticks : 3858318
TotalDays : 4.46564583333333E-06
TotalHours : 0.0001071755
TotalMinutes : 0.00643053
TotalSeconds : 0.3858318
TotalMilliseconds : 385.8318
# Standard ForEach-Object
Measure-Command {
get-appxpackage -allusers |
ForEach-Object {select name}
}
# Results
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
*** Milliseconds : 498 ***
Ticks : 4988494
TotalDays : 5.77371990740741E-06
TotalHours : 0.000138569277777778
TotalMinutes : 0.00831415666666667
TotalSeconds : 0.4988494
TotalMilliseconds : 498.8494
IIS Version: 8.5.9600.16384
PowerShell version: 4.0
I am trying to get the periodic restart recycle value of a specific apppool like below :
$b=(Get-ItemProperty 'IIS:\AppPools\testapppool' -Name `
Recycling.periodicRestart.schedule.collection).value
$b
It is giving me output like:
Days : 0
Hours : 5
Minutes : 36
Seconds : 0
Milliseconds : 0
Ticks : 201600000000
TotalDays : 0.233333333333333
TotalHours : 5.6
TotalMinutes : 336
TotalSeconds : 20160
TotalMilliseconds : 20160000
If I run:
Get-ItemProperty 'IIS:\AppPools\testapppool' -Name`
Recycling.periodicRestart.schedule.collection
I get output like:
value : 05:36:00
Attributes : {value}
ChildElements : {}
ElementTagName : add
Methods :
Schema : Microsoft.IIs.PowerShell.Framework.ConfigurationElementSchema
Again when I run below snippet, it give same value output :
[System.Reflection.Assembly]::LoadFrom( `
"C:\windows\system32\inetsrv\Microsoft.Web.Administration.dll" )
$serverManager = new-object Microsoft.Web.Administration.ServerManager
$b=$serverManager.ApplicationPools |where {$_.name -eq 'tetsapppool'}
$var=$b.Recycling.PeriodicRestart.schedule |select time
$var
Time
----
05:36:00
Please help me figuring out a way to assign only the value of the periodic restart recycle value to another variable so that, I can compare it and operate upon the value of the variable.
I have a huge array which contains dates. The date has the following form: tt.mm.yyyy. I know how to sort the array with Sort-Object, but the sorting takes a lot of time. I found another way of sorting arrays, but it doesn't work as expected.
My former code to sort the array was like this.
$data | Sort-Object { [System.DateTime]::ParseExact($_, "dd.MM.yyyy", $null) }
But as I siad before: this way of sorting is too slow. The Sort() method from System.Array seems to be much faster.
[Array]::Sort([array]$array)
This code sorts an array containing strings much faster than Sort-Object. Is there a way how I can change the above sorting method like the Sort-Object method?
The .NET method will work for dates if you make sure that the array is of type DateTime.
Meaning you should use
[DateTime[]]$dateArray
instead of
[Array]$dateArray
when you create it. Then you can use
[Array]::Sort($dateArray)
to perform the sort it self...
Your input data are date strings with a date format that doesn't allow sorting in "date" order. You must convert the strings either to actual dates
Get-Date $_
[DateTime]::ParseExact($_, "dd.MM.yyyy", $null)
or change the format of the string dates to ISO format, which does allow sorting in date order.
'{2}-{1}-{0}' -f ($_ -split '.')
'{0}-{1}-{2}' -f $_.Substring(6,4), $_.Substring(3,2), $_.Substring(0,2)
$_ -replace '(\d+)\.(\d+).(\d+)', '$3-$2-$1'
At some point you must do one of these conversions, either when creating the data or when sorting.
I ran some tests WRT performance of each conversion, and string transformation using the Substring() method seems to be the fastest way:
PS C:\> $dates = 1..10000 | % {
>> $day = Get-Random -Min 1 -Max 28
>> $month = (Get-Random -Min 1 -Max 12
>> $year = Get-Random -Min 1900 -Max 2014
>> '{0:d2}.{1:d2}.{2}' -f $day, $month, $year
>> }
>>
PS C:\> Measure-Command { $dates | sort {Get-Date $_} }
Days : 0
Hours : 0
Minutes : 0
Seconds : 1
Milliseconds : 520
Ticks : 15200396
TotalDays : 1,75930509259259E-05
TotalHours : 0,000422233222222222
TotalMinutes : 0,0253339933333333
TotalSeconds : 1,5200396
TotalMilliseconds : 1520,0396
PS C:\> Measure-Command { $dates | sort {'{2}-{1}-{0}' -f ($_ -split '.')} }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 413
Ticks : 4139027
TotalDays : 4,79054050925926E-06
TotalHours : 0,000114972972222222
TotalMinutes : 0,00689837833333333
TotalSeconds : 0,4139027
TotalMilliseconds : 413,9027
PS C:\> Measure-Command { $dates | sort {$_ -replace '(\d+)\.(\d+).(\d+)', '$3-$2-$1'} }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 348
Ticks : 3488962
TotalDays : 4,03815046296296E-06
TotalHours : 9,69156111111111E-05
TotalMinutes : 0,00581493666666667
TotalSeconds : 0,3488962
TotalMilliseconds : 348,8962
PS C:\> Measure-Command { $dates | sort {[DateTime]::ParseExact($_, "dd.MM.yyyy", $null)} }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 340
Ticks : 3408966
TotalDays : 3,9455625E-06
TotalHours : 9,46935E-05
TotalMinutes : 0,00568161
TotalSeconds : 0,3408966
TotalMilliseconds : 340,8966
PS C:\> Measure-Command { $dates | sort {'{0}-{1}-{2}' -f $_.Substring(6,4), $_.Substring(3,2), $_.Substring(0,2)} }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 292
Ticks : 2926835
TotalDays : 3,38754050925926E-06
TotalHours : 8,13009722222222E-05
TotalMinutes : 0,00487805833333333
TotalSeconds : 0,2926835
TotalMilliseconds : 292,6835
edit: Should I have posted this on serverfault instead? There is not even a dfs-r category on stackoverflow, but I thought this was more of a scripting\programming question. Let me know if I should put this on serverfault instead.
I attempted to use the DfsrIdRecordInfo class to retrieve all files of a somewhat large (6000 file) DFSR database and was getting WMI quota errors.
Doubling and even tripling the wmi quotas on the server did not solve this.
I found looking here that the index property of this class is: "The run-time index of the record. This value is used to partition the result of a large query." which sounded like exactly what I wanted, but the behavior of this property is not what I expected.
I found that when I try to do paging with this property it does not retrieve all of the records as per the following example with powershell.
I tested this on a DFSR database with less than 700 files that does not throw a quota error. Because this is a small database I can get all the files like this in less than a second:
$DFSRFiles =
gwmi `
-Namespace root\microsoftdfs `
-ComputerName 'dfsrserver' `
-Query "SELECT *
FROM DfsrIdRecordInfo
WHERE replicatedfolderguid = '$guid'"
PS F:\> $DFSRFiles.count
680
So I have 680 files in this DFSR DB. Now if I try to use the index property for pagination like this:
$starttime = Get-Date;
$i = 0 #index counter
$DfsrIdRecordInfoArr = #()
while ($i -lt 1000) {
$starttimepage = Get-Date
$StartRange = $i
$EndRange = $i += 500
Write-Host -ForegroundColor Green "On range: $StartRange - $EndRange"
$DFSRFiles =
gwmi `
-Namespace root\microsoftdfs `
-ComputerName 'dfsrserver' `
-Query "SELECT *
FROM DfsrIdRecordInfo
WHERE index >= $StartRange
AND index <= $EndRange
AND replicatedfolderguid = '$guid'"
$DfsrIdRecordInfoArr += $DFSRFiles
Write-Host -ForegroundColor Green "Returned $($DFSRFiles.count) objects from range"
(Get-Date) - $starttimepage
write-host -fo yellow "DEBUG: i = $i"
}
(get-date) - $starttime
PS F:\> $DfsrIdRecordInfoArr.count
517
So it only returns 517 files.
Here is the full output of my debug messages. You can also see searching this way takes a super long time:
On range: 0 - 500
Returned 501 objects from range
Days : 0
Hours : 0
Minutes : 1
Seconds : 29
Milliseconds : 540
Ticks : 895409532
TotalDays : 0.001036353625
TotalHours : 0.024872487
TotalMinutes : 1.49234922
TotalSeconds : 89.5409532
TotalMilliseconds : 89540.9532
DEBUG: i = 500
On range: 500 - 1000
Returned 16 objects from range
Days : 0
Hours : 0
Minutes : 1
Seconds : 35
Milliseconds : 856
Ticks : 958565847
TotalDays : 0.00110945121180556
TotalHours : 0.0266268290833333
TotalMinutes : 1.597609745
TotalSeconds : 95.8565847
TotalMilliseconds : 95856.5847
DEBUG: i = 1000
Days : 0
Hours : 0
Minutes : 3
Seconds : 5
Milliseconds : 429
Ticks : 1854295411
TotalDays : 0.00214617524421296
TotalHours : 0.0515082058611111
TotalMinutes : 3.09049235166667
TotalSeconds : 185.4295411
TotalMilliseconds : 185429.5411
Am I doing something stupid? I was thinking that "run-time index" means the index property is not statically attached to the records and is generated anew for each record every time a query is run because the index properties of objects in $DFSRFiles do not match those in $DfsrIdRecordInfoArr.
But if the index property is different for every query then I would have duplicates in $DfsrIdRecordInfoArr which I do not. All the records are unique, but it just doesn't return all of them.
Is the index property totally useless for my purpose? Perhaps when it says "...partition the result of a large query" this means it is to be used on records that have already been returned from WMI not the WMI query itself.
Any guidance would be appreciated. Thanks in advance.