I am trying to aggregate some statistics from my broker. I have raw data in CSV format. Here are several strings from the CSV:
"Date";"OperationType";"InstrumentType";"Payment";"Currency"
"30.07.2021 6:00:00";"Dividend";"Stock";"3,42";"USD"
"30.07.2021 6:00:00";"Dividend";"Stock";"0,16";"USD"
"29.07.2021 13:55:15";"BrokerCommission";"Currency";"-7,32";"RUB"
"29.07.2021 13:55:14";"Buy";"Currency";"-14635,5";"RUB"
"29.07.2021 13:55:13";"PayIn";;"14642,82";"RUB"
"29.07.2021 6:00:00";"Dividend";"Stock";"1,93";"USD"
"29.07.2021 6:00:00";"Dividend";"Stock";"1107";"RUB"
"29.07.2021 6:00:00";"TaxDividend";"Stock";"-144";"RUB"
"28.07.2021 12:13:18";"BrokerCommission";"Currency";"-7,34";"RUB"
"28.07.2021 12:13:17";"Buy";"Currency";"-14683";"RUB"
"28.07.2021 12:13:17";"PayIn";;"14690,35";"RUB"
"28.07.2021 12:12:38";"BrokerCommission";"Stock";"-0,1";"USD"
"28.07.2021 12:12:37";"Buy";"Stock";"-196,71";"USD"
"28.07.2021 7:58:17";"BrokerCommission";"Currency";"-3,68";"RUB"
"28.07.2021 7:58:16";"Buy";"Currency";"-7369,75";"RUB"
"28.07.2021 7:58:15";"PayIn";;"7373,44";"RUB"
"28.07.2021 0:35:08";"BrokerCommission";"Stock";"-0,06";"USD"
"28.07.2021 0:35:07";"Buy";"Stock";"-122,23";"USD"
"28.07.2021 0:34:16";"BrokerCommission";"Stock";"-0,14";"USD"
"28.07.2021 0:34:15";"Buy";"Stock";"-278,92";"USD"
"28.07.2021 0:33:18";"BrokerCommission";"Stock";"-0,07";"USD"
"28.07.2021 0:33:17";"Buy";"Stock";"-142,76";"USD"
"28.07.2021 0:32:31";"BrokerCommission";"Stock";"-0,04";"USD"
"28.07.2021 0:32:30";"Buy";"Stock";"-84,31";"USD"
Here is the code I use for it:
#Initiate arrays
$InputArray = #()
$FinalArray = #()
$GroupedArray = #()
#Create array with source data
$InputArray = Import-CSV "path_to_csv_file\file.csv" -Delimiter ";" -Encoding UTF8 | Where-Object { $_.PSObject.Properties.Value -ne '' }
#Convert strings to appopriate data types
foreach ($Object in $InputArray){
$Object.Date = [datetime]::parse($Object.Date) | Get-Date -Format "yyyy-MM-dd"
$Object.Payment = [double]($Object.Payment -replace(',','.'))
}#foreach
#Group objects
$GroupedArray = $InputArray | Group-Object -Property "Date","Currency","OperationType" | `
Select-Object #{Name='Date' ;Expression={$_.Values[0]}}, `
#{Name='Currency' ;Expression={$_.Values[1]}}, `
#{Name='OperationType' ;Expression={$_.Values[2]}}, `
#{Name='Payment' ;Expression={($_.Group | Measure-Object 'Payment' -Sum).Sum}} | `
Group-Object Date
foreach ($Object in $GroupedArray){
$Date = $Object.Name
foreach ($Instance in $Object.Group){
if ($Instance.OperationType = "BrokerCommission" -and $Instance.Currency -eq "RUB"){
$BrokerComissionRUB = $null
$BrokerComissionRUB = $Instance.Payment
}#if
if ($Instance.OperationType = "BrokerCommission" -and $Instance.Currency -eq "USD"){
$BrokerComissionUSD = $null
$BrokerComissionUSD = $Instance.Payment
}#If
if ($Instance.OperationType = "Dividend" -and $Instance.Currency -eq "RUB"){
$DividendRUB = $null
$DividendRUB = $Instance.Payment
}#if
if ($Instance.OperationType = "Dividend" -and $Instance.Currency -eq "USD"){
$DividendUSD = $null
$DividendUSD = $Instance.Payment
}#If
if ($Instance.OperationType = "PayIn" -and $Instance.Currency -eq "RUB"){
$PayInRUB = $null
$PayInRUB = $Instance.Payment
}#if
}#foreach
$FinalArray += [PSCustomObject]#{
"Date" = $Date
"PayInRUB" = $PayInRUB
"DividendRUB" = $DividendRUB
"DividendUSD" = $DividendUSD
"BrokerComissionRUB" = $BrokerComissionRUB
"BrokerComissionUSD" = $BrokerComissionUSD
}
}#foreach
$FinalArray
The problem is that in $InputArray value of, for example $InputArrary[1].OperationType, has STRING format:
$InputArray[1].OperationType | gm
TypeName: System.String
But right after grouping objects with Group-Object, OperationType value type is changed to Boolean:
$GroupedArray[1].Group.operationtype | gm
TypeName: System.Boolean
So because of this strange type transformation the actual data is lost.
Nevertheless if I execute a part of a script before foreach cycle, the data type of OperationType is still STRING in $GroupedArray. If I run whole script, it changes to Boolean.
I have no idea why is that haapening and how can I avoid this. Could you please advice?
There's no strange transformation, You just have a typo, you need to use -eq and not =
You are actually set the $Instance.OperationType value to "BrokerCommission" and then replace it's value again with the results of $Instance.Currency -eq "RUB" which is a boolean value
Check it yourself in the console, execute this:
$Instance.OperationType = "BrokerCommission" -and $Instance.Currency -eq "RUB"
Results:
$Instance.OperationType
False
Related
I'm new to scripting so please excuse me if my script is messy. This script pretty much does what I want it to do but for 2 fields it doesn't return the values.
If I run the commands without Invoke I get all the values I want but when I run this with the Invoke command on remote computers the OsHotFixes and CsProcessors return weird values of "Microsoft.PowerShell.Commands.HotFix" for each hotfix and "Microsoft.PowerShell.Commands.Processor" for the CsProcessors value. All other properties gave me the values I am looking for. I'm not sure why those 2 aren't returning correct values. If someone could point me in the right direction that would be awesome.
$c = Get-Content "myfilepath"
$e = "myfilepath"
$ScriptBlock = {
$ComputerInfo = Get-ComputerInfo -Property WindowsVersion, OsBuildNumber, OsHotFixes, CsModel, BiosSMBIOSBIOSVersion, WindowsProductName, CsProcessor, OsInstallDate, OsArchitecture, CsProcessors
$GPU = Get-WmiObject win32_VideoController | Select-Object "Name", "DeviceID", "DriverVersion"
$RAM = Get-CimInstance -ClassName CIM_PhysicalMemory | Select-Object "Manufacturer", "PartNumber", #{'Name'='Capacity (GB)'; 'Expression'={[math]::Truncate($_.capacity / 1GB)}}, "Speed"
$Storage = Get-WmiObject Win32_LogicalDisk | Where caption -eq "C:" | Foreach-object {write " $($_.caption) $('{0:N2}' -f ($_.Size/1gb)) GB total, $('{0:N2}' -f ($_.FreeSpace/1gb)) GB Free"}
$MyArray = #($ComputerInfo, $GPU, $RAM, $Storage)
$Properties =
#(
'WindowsVersion'
'OsBuildNumber'
'OsHotFixes'
'CsModel'
'BiosSMBIOSBIOSVersion'
'WindowsProductName'
'OsInstallDate'
'OsArchitecture'
'CsProcessors'
'Name'
'DeviceID'
'DriverVersion'
'Manufacturer'
'PartNumber'
'Capacity'
'Speed'
'Disk'
)
$MyArray | ForEach-Object {
:Inner ForEach( $Property in $Properties )
{
If($_.$Property)
{
[PSCustomObject][Ordered]#{
hostname = $env:COMPUTERNAME
WindowsVersion = $_.WindowsVersion
Build = $_.OsBuildNumber
Patches = $_.OsHotFixes
Motherboard = $_.CsModel
BiosVersion = $_.BiosSMBIOSBIOSVersion
WindowsProductName = $_.WindowsProductName
OsInstallDate = $_.OsInstallDate
OsArchitecture = $_.OsArchitecture
Processor = $_.CsProcessors
GPUName = $_.Name
DeviceID = $_.DeviceID
DriverVersion = $_.DriverVersion
RamManufacturer = $_.Manufacturer
PartNumber = $_.PartNumber
Capacity = $_.Capacity
Speed = $_.Speed
Disk = $Storage
}
Break Inner
}
}
}
}
Invoke-Command -ComputerName $c -ScriptBlock $ScriptBlock | Sort hostname | Export-Csv -append $e -NoTypeInformation
I've tried running just the lines from 4 - 8 locally and then Outputting the Array. This will show all correct values. However when this script runs with the PSCustomObject and Invoke command I don't get CsProcessors or OsHotFixes values.
I have two arrays $list_CloudUsers and $list_Active. Both have a column called Alias.
I want to filter $list_CloudUsers so that the list does not have any of the ALIASes that are contained in $list_Active.
I was able to do it with:
$list_arc = Import-CSV $Arc_LastAccess
$list_Active = $list_arc | Where { [int]$_.InactivityDays -le 30}
$NewList = #()
ForEach ($User_Cloud in $list_CloudUsers)
{
$Alias = $User_Cloud.Alias
if ($list_Active -match $alias) {continue}
$NewList += $User_Cloud
}
But it does not work with this. Any ideas how I can get the WHERE to work correctly.
$list_arc = Import-CSV $Arc_LastAccess
$list_Active = $list_arc | Where { [int]$_.InactivityDays -le 30}
$NewList1 = $list_CloudUsers | Where {$list_Active -NotMatch $_.alias}
Try
$NewList1 = $list_CloudUsers | Where-Object {$list_Active.Alias -notcontains $_.Alias}
$allResources = #()
$subscriptions=Get-AzSubscription
ForEach ($vsub in $subscriptions){
Select-AzSubscription $vsub.SubscriptionID
Write-Host
Write-Host "Working on "$vsub
Write-Host
$allResources += $allResources |Select-Object $vsub.SubscriptionID,$vsub.Name
$result=#()
$webapps = Get-AzWebApp
foreach($webapp in $webapps){
$Tier = (Get-AzResource -ResourceId $webapp.ServerFarmId).Sku.Tier
$SKU = (Get-AzAppServicePlan -ResourceGroupName $webapp.ResourceGroup).Sku.Size
$AppServiceName = (Get-AzAppServicePlan -ResourceGroupName $webapp.ResourceGroup).Name
$obj = [PSCustomObject]#{
TenantId = $vsub.TenantId
SubscriptionName = $vsub.Name
WebappName = $webapp.Name
ResourceGroup = $webapp.ResourceGroup
Hostname = $WebApp.DefaultHostName
PricingTier = $Tier
SKU = ($SKU -join ',')
AppServiceName = ($AppServiceName -join ',')
#State = $webapp.State
#Location = $webapp.Location
#AppType = $webapp.Kind
}
$result += $obj
$result | Export-Csv -Path "E:\webapps_filter.csv" -Append -NoTypeInformation
$input = 'E:\webapps_filter.csv'
$inputCsv = Import-Csv $input | Sort-Object * -Unique
$inputCsv | Export-Csv "E:\webapps.csv" -NoTypeInformation}}
Right now I am using the above script to fetch all the required data of web apps from all the subscriptions. Currently, the script is taking time to execute, I need to optimize it and also the script gives a duplicate output so in last have added the filter to sort out it by unique entry.
It seems you are adding stuff to an array variable $allResources you don't use, so get rid of that.
Instead of the costly (time/memory) $result += $obj, better let PowerShell collect the objects using $result = foreach(..)
You are appending a temporary file inside the foreach loop on each iteration, then import this file and filter it on all properties to become unique.
again, inside the loop you are exporting this uniqified data to a CSV file on every iteration
You are calling upon cmdlet Get-AzAppServicePlan twice to get different properties. Use it only once would save time
as aside, you should not use a self-defined variable called $input as this is an Automatic variable
Try:
$subscriptions = Get-AzSubscription
$result = foreach ($vsub in $subscriptions){
Select-AzSubscription $vsub.SubscriptionID
Write-Host
Write-Host "Working on $($vsub.Name)"
Write-Host
foreach($webapp in (Get-AzWebApp)){
$Tier = (Get-AzResource -ResourceId $webapp.ServerFarmId).Sku.Tier
$Plan = Get-AzAppServicePlan -ResourceGroupName $webapp.ResourceGroup
# output the object so it gets collected in $result
[PSCustomObject]#{
TenantId = $vsub.TenantId
SubscriptionName = $vsub.Name
SubscriptionID = $vsub.SubscriptionID
WebappName = $webapp.Name
ResourceGroup = $webapp.ResourceGroup
Hostname = $webapp.DefaultHostName
PricingTier = $Tier
SKU = #($Plan.Sku.Size) -join ','
AppServiceName = #($Plan.Name) -join ','
#State = $webapp.State
#Location = $webapp.Location
#AppType = $webapp.Kind
}
}
}
# sort unique and export the file
$result | Sort-Object * -Unique | Export-Csv -Path "E:\webapps.csv" -NoTypeInformation
I'm creating an array of PSObjects with calculated properties. I need one property that is calculated based on another property of the same object. How do I do that?
Example - let's say I have array of strings like "a_1", "b_2", "c_3" etc. and I have a lookup function that returns something based on the first part of those strings, i.e. someLookUpFunction('a') would return "AA" with input of "a".
Now I need a property in my object that has this calculated 'AA' based on the my 'name' property
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray | ForEach-Object{
New-Object PSObject -Property #{
'name' = ($_ -split "_")[0]
'extendedName' = {$name = ($_ -split "_")[0]; someLookUpFunction($name) }
}
}
The code above doesn't work in part that the output for 'extendedName' property is just this script block. How do I make it to take the value?
If you need to capture the output of an expression within an expression, you can use the sub-expression operator $().
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray | ForEach-Object {
[pscustomobject]#{
'name' = ($_ -split "_")[0]
# You can't reference the name property above in this property because it has not been created yet.
'extendedName' = $($name = ($_ -split "_")[0]; someLookUpFunction $name)
}
}
However, that should not be necessary in your example. You can define a variable before the custom object creation and then reference it within the object creation code:
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray | ForEach-Object {
$name = ($_ -split '_')[0]
[pscustomobject]#{
'name' = $name
'extendedName' = someLookUpFunction $name
}
}
You could also pass expressions to parameters directly provided it can be tokenized correctly:
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray | ForEach-Object {
[pscustomobject]#{
'name' = ($_ -split '_')[0]
'extendedName' = someLookUpFunction ($_ -split '_')[0]
}
}
Note: The proper way to call a function without using the pipeline is functionName -parametername parametervalue or functionName parametervalue if positional parameters are enabled. The syntax functionName(parametervalue) could have unintended consequences. See this answer for a deeper dive into function/method calling syntax.
You cannot access the name property of an object before that object has been created.
In addition to AdminOfThings Good Answer you can bypass the loop altogether using a select statement with the calculated property hash syntax:
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray |
Select-Object #{Name = 'Name'; Expression = { ($_ -Split '_')[0] } },
#{Name = 'ExtendedName'; Expression = { SomeLookupFunction ($_ -Split '_')[0] } }
For the efficiency of not executing -Split '_' 2x, if you do go with a loop just use a variable to and reference twice.
Altered version of AdminOfThings Example:
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray | ForEach-Object {
$TmpName = ($_ -split '_')[0]
[pscustomobject]#{
'name' = $TmpName
'extendedName' = someLookUpFunction $TmpName
}
}
It's also correct that you can't reference a property before it's been added to an object. One way around this is to just use 2 select statements:
$stringArray = #('a_1', 'b_2', 'c_3')
$objectArray = $stringArray |
Select-Object #{Name = 'Name'; Expression = { ($_ -Split '_')[0] } } |
Select-Object *, #{Name = 'ExtendedName'; Expression = { SomeLookupFunction ($_ -Split '_')[0] } }
This may have some readability advantage, but, I try to avoid it in favor of invoking as few commands as possible.
ive been dabbling with powershell for a while now and ive been trying to modify some data in an array.
Problem is that my source array is very large and this script takes hours to run. Maybe someone can help my optimize my script.
With a small source array the script runs just fine btw.
$array_metric_hour = #()
$array_metric_hour =
foreach ($resource in $resources) {
Write-Progress -Id 0 "Step $resource"
foreach ($hour in $Time_Array) {
Write-Progress -Id 1 -ParentId 0 "Step $resource - Substep" ($hour.timestamp+":00")
[pscustomobject] #{
resourceID = $resource
resourceName = $array_bill.resources.($resource).name
time = $hour.timestamp+":00"
Poweredon = ((($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp}).poweredon | Measure-Object -Maximum).Maximum)
#Cpu_On = if (($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -eq "0,0"}).poweredon) {0} else {(($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -ne "0,0"}).provisionedCpu | Measure-Object -Maximum).Maximum}
Mem_GB_On = if (($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -eq "0,0"}).poweredon) {0} else {(($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp -and $_.poweredOn -ne "0,0"}).provisionedMem_GB | Measure-Object -Maximum).Maximum}
hardware_Diskspace_GB = ((($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp}).hardware_Diskspace_GB | Measure-Object -Maximum).Maximum)
#used_Diskspace_GB = ((($Array_combined | Where-Object {$_.resourceID -eq $resource -and $_.hour -eq $hour.timestamp}).used_Diskspace_GB | Measure-Object -Maximum).Maximum)
}
}
}
Some extra information that is required:
$Time_array has every full hour in a month, so 745 values in this case.
$array_combined exist of 98131 lines (5 minute interval with metrics during a month.
this array has the folowing items per interval.
resourceID
resourceName
timestamps
human_timestamp
hour
date
poweredOn
provisionedMem_GB
hardware_Diskspace_GB
used_Diskspace_GB
thanks for all the comments, next time ill try to supply all and correct information.
The suggestions of creating an extra filter was the winner for me. the scripts is 50 times faster in the current state and for now quick enough.
Added $filter1 and $filter2.
$array_metric_hour = #()
$array_metric_hour =
foreach ($resource in $resources) {
$filter1 = $Array_combined | Where-Object {$_.resourceID -eq $resource}
Write-Progress -Id 0 "Step $resource"
foreach ($hour in $Time_Array) {
$filter2 = $filter1 | Where-Object {$_.hour -eq $hour.timestamp}
Write-Progress -Id 1 -ParentId 0 "Step $resource - Substep" ($hour.timestamp+":00")
[pscustomobject] #{
resourceID = $resource
resourceName = $array_bill.resources.($resource).name
time = $hour.timestamp+":00"
Poweredon = if ($filter2 | where poweredOn -eq "0,0") {"0"} else {($filter2.poweredOn | Measure-Object -Maximum).Maximum}
Mem_GB_On = if ($filter2 | where poweredOn -eq "0,0") {0} else {(($filter2).provisionedMem_GB | Measure-Object -Average).Average}
hardware_Diskspace_GB = ((($filter2).hardware_Diskspace_GB | Measure-Object -Average).Average)
}
}
}
Since you're still not have provided much more of your code (e.g. where does "Array_combined" come from?), here are some important notes:
Don't use "Write-Progress" on every iteration! It has a very very huge impact on performace when using PS <=5.1, 6 and 7.
With the current "7.1" build I am using it works like a charm ("7.1.0-preview.7"). Have to look when they fixed it.
Avoid pipe'ing as much as you can when you want to have the best performance. Streaming data from one command to the other is compared to statements like "foreach {}" (NOT "Foreach-Object"!) really bad.
Here is an example for your template, even when there are some important steps missing:
# Progress bar definition
$progressActivity1 = 'Processing items'
$progressCounter1 = -1
$progressMax1 = #($resources).Count
$progressInterval1 = [math]::Ceiling($progressMax1 * 0.1) # each 10%
$progressId1 = 1
$progressParentId1 = 0
# *** use a list if your script adds objects several times.
# *** Note: "Arrays" are immutable and will be re-created each time you add something
$array_metric_hour = [System.Collections.Generic.List[psobject]]::new()
# *** good approach to add the result of a forEach-statement directly to variable. Performance is similar compared to adding objects to a list.
$array_metric_hour = foreach ($resource in $resources) {
# Progress bar counter & drawing (each 10%)
$progressCounter1++
If ($progressCounter1 % $progressInterval1 -eq 0) {
Write-Progress -Activity $progressActivity1 -PercentComplete($progressCounter1 / $progressMax1 * 100) -Id $progressId1 -ParentId $progressParentId1
}
# *** "Array_combined" is unknwon.... but according to the usage:
# !!! try to create a dictionary/hashtable of "Array_combined" with "resourceID" as key.
# !!! hash tables/dictionaries are much faster to access a particular item than arrays
# !!! access would be: $filter1 = $Array_combined[$resource]
$filter1 = $Array_combined | Where-Object { $_.resourceID -eq $resource }
# Progress bar definition
$progressActivity2 = 'Processing items'
$progressCounter2 = -1
$progressMax2 = #($Time_Array).Count
$progressInterval2 = [math]::Ceiling($progressMax2 * 0.1) # each 10%
$progressId2 = 2
$progressParentId2 = $progressId1
foreach ($hour in $Time_Array) {
# ??? don't know what $filter1 is about ...
# !!! replace that; use a hastable/dictionary
# !!! alternatively: use "foreach"-statement OR method ".where{}" which was introduced in PS 4.0
$filter2 = $filter1 | Where-Object { $_.hour -eq $hour.timestamp }
# Progress bar counter & drawing (each 10%)
$progressCounter2++
If ($progressCounter2 % $progressInterval2 -eq 0) {
Write-Progress -Activity $progressActivity2 -PercentComplete($progressCounter2 / $progressMax2 * 100) -Id $progressId2 -ParentId $progressParentId2
}
[pscustomobject] #{
resourceID = $resource
resourceName = $array_bill.resources.($resource).name
time = $hour.timestamp + ':00'
# ??? "Where-Object" could be replaced ... but don't know the background or data ....
# !!! replace "Measure-Object" with "[Linq.Enumerable]" methods if possible
Poweredon = if ($filter2 | Where-Object poweredOn -EQ '0,0') { '0' } else { ($filter2.poweredOn | Measure-Object -Maximum).Maximum }
# !!! same as above
Mem_GB_On = if ($filter2 | Where-Object poweredOn -EQ '0,0') { 0 } else { (($filter2).provisionedMem_GB | Measure-Object -Average).Average }
# !!! same as above
hardware_Diskspace_GB = ((($filter2).hardware_Diskspace_GB | Measure-Object -Average).Average)
}
}
}
# Progress completed
Write-Progress -Activity $progressActivity -Completed -Id $progressId1
Write-Progress -Activity $progressActivity -Completed -Id $progressId2