Masking fields duplications - powershell

I am trying to mask fields in a string as seen below. It is working to an extent, half way really. At some stage after the $addresspostcode the replacement characters aren't replacing for the correct positions. Would anyone have an idea of a fix?
The adressee0 line is from the output file
ADDRESSEE0|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY|YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY|YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY|YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY|ZZZZZZZZ|Sir or MadamZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ |A1|OM|Mr Patrick MurphyZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ |XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|45 CregtownZZZZZZZZZZZZZZZZ |EastRoad RoadZZZZZZZZZZZZZZZZ |TownnamersZZZZZZZZZZZZZZZZ |CityAB 16ZZZZZZZZZZZZZZZZ |ZZZZZZZZ| |XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|ZZZZZZZZZZZZZZZZ |Sir or MadamZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ |IA|3319041| | |XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXZZZZZZZZ
ForEach-Object {
$addresseeName = $_.Substring(11,50)
$addresseeName2 = $_.Substring(62,50)
$addresseeLine1 = $_.Substring(113,30)
$addresseeLine2 = $_.Substring(144,30)
$addresseeLine3 = $_.Substring(175,30)
$addresseeLine4 = $_.Substring(206,30)
$addresseePostCode = $_.Substring(237,8)
$referenceAddressName1 = $_.Substring(303,50)
$referenceAddressName2 = $_.Substring(354,50)
$referenceAddresseeLine1 = $_.Substring(405,30)
$referenceAddresseeLine2 = $_.Substring(436,30)
$referenceAddresseeLine3 = $_.Substring(467,30)
$referenceAddresseeLine4 = $_.Substring(498,30)
$mask50 = 'X' * 50
$mask30 = 'Y' * 30
$mask08 = 'Z' * 8
# IF statement, if the string is at position 0-10, and begins with 'ADDRESSEE0'
# then run replace statement
if ($_.Substring(0,10) -eq 'ADDRESSEE0') {
$_.Replace($addresseeName, $mask50).Replace($addresseeName2, $mask50).Replace($addresseeLine1, $mask30).Replace($addresseeLine2, $mask30).Replace($addresseeLine3, $mask30).Replace($addresseeLine4, $mask30).Replace($addresseePostCode, $mask08).Replace($referenceAddressName1, $mask50).Replace($referenceAddressName2, $mask50).Replace($referenceAddresseeLine1, $mask30).Replace($referenceAddresseeLine2, $mask30).Replace($referenceAddresseeLine3, $mask30).Replace($referenceAddresseeLine4, $mask30)

The problem with your code is that it does the replace based on the contents of the string (rather than the position). This means if the same text exists elsewhere in the string, it replaces that text also, breaking the later replaces.
I suggest you do this instead:
$mask50 = 'X' * 50
$mask30 = 'Y' * 30
$mask08 = 'Z' * 8
$SomeInput | ForEach-Object {
if ($_.Substring(0,10) -eq 'ADDRESSEE0')
{
$SplitString = $_.Split('|')
1..2 | ForEach-Object { $SplitString[$_] = $mask50 }
3..6 | ForEach-Object { $SplitString[$_] = $mask30 }
$SplitString[7] = $mask08
8..9 | ForEach-Object { $SplitString[$_] = $mask50 }
10..13 | ForEach-Object { $SplitString[$_] = $mask30 }
$SplitString -Join '|'
}
}
This splits the string based on the | character and then does individual replaces for each (we use the .. array notation to make this a little more efficient).
Then we join the string again at the end with the | character.

Assuming ADDRESSEE0|... is the input string, why not split the data first? This provides manageable chunks instead of one giant string with truckload of method chaining. Like so,
# Get input data
$raw = `'ADDRESSEE0|ADDRESSEE1|ADDRESSEELine0|ADDRESSEELine1|...'`
# Split the string by each pipe | char. This uses regex syntax, so escape \ is needed
$lines = $raw -split '\|'
# Assign splitted elements into more readable variables
$addresseeName = $lines[0]
$addresseeName2 = $lines[1]
$addresseeLine1 = $lines[2]
...
# mask the data whatever way floats your boat
$addresseeName = $addresseeName.substring(0,9) + $mask08

Related

Sort a nested hash table by value

I have a hash array as such:
$weeklyStats[$RecipientName][$weekNr][$total]
Which is created in a loop as such:
$weeklyStats = #{}
$weekNr = get-date -UFormat %V
ForEach ($RecipientName in $MailTraffic.keys)
{
$weeklyStats[$RecipientName] = #{}
$weeklyStats[$RecipientName][$weekNr] = #{}
$weeklyStats[$RecipientName][$weekNr]['Total'] = 0
$weeklyStats[$RecipientName][$weekNr]['Sent'] = 0
$weeklyStats[$RecipientName][$weekNr]['Received'] = 0
foreach($item in $MailTraffic[$RecipientName].keys)
{
weeklyStats[$RecipientName][$weekNr]['Total'] =+ 1
if $MailTraffic[$RecipientName]['transaction'] == "Sent"
{
$weeklyStats[$RecipientName][$weekNr]['Sent'] =+ 1
}
else
{
$weeklyStats[$RecipientName][$weekNr]['Received'] =+ 1
}
}
}
I don't know how to 'dump' a variable in Powershell but here is the contents in json:
{
"mike": {
"11": {
"Total": 411,
"Sent": 21,
"Received":390,
}
},
"peter": {
"11": {
"Total": 751,
"Sent": 51,
"Received":700,
}
},
"frank": {
"11": {
"Total": 620,
"Sent": 20,
"Received":600,
}
},
}
I want to print out the keys and values in descending order of the $total.
I can only find examples how to do it if the hash table is only one level deep.
The intended output would be:
Name Total Received Sent
----- ----- ----- -----
peter 751 700 51
frank 620 600 20
mike 411 390 21
Sort by referencing the Keys property of the inner hashtable, then assign to a new [ordered] dictionary:
$sorted = [ordered]#{}
$stats.GetEnumerator() |Sort-Object {
# Sort by first key from each inner hashtable
$_.Value.Keys |Select -First 1
} -Descending |ForEach-Object {
# re-assign to our ordered dictionary
$sorted[$_.Key] = $_.Value
}
$sorted now contains your new sorted dictionary
Most PowerShell cmdlets are intended to handle (stream!) [PSObject] type (which includes a [PScustomerObject] type) lists for input and output.
(To understand the difference see e.g. Difference between PSObject, Hashtable, and PSCustomObject).
Nested hash tables are difficult to maintain and handle in PowerShell (see also: Powershell Multidimensional Arrays) because PowerShell is optimized for streaming which is rather difficult with cascaded objects, therefore I recommend you convert you nested hashtable in a (rather flat) [PScustomerObject] list, something like:
$PSStats =
ForEach ($name in $Stats.Keys) {
ForEach ($weekNr in $_.Keys) {
ForEach ($total in $_.Keys) {
[pscustomobject]#{name = $name; weekNr = $weekNr; total = $total}
}
}
}
Once you have converted it into PSCustomObject list, you can easily sort it and display the results:
$PSStats | Sort-Object Total
I would just create a custom object from your hash tables and then sort on the Title property:
# Creation of $mailtraffic
$mailtraffic = #{'Mike' = #{'Total' = 411; 'Sent' = 21; 'Received' = 390};'Peter' = #{'Total' = 751; 'Sent' = 51; 'Received' = 700};'Frank' = #{'Total' = 620; 'Sent' = 20; 'Received' = 600}}
# Sorting Code
$mailtraffic.GetEnumerator() |
Select #{n='Name';e={$_.Key}},#{n='Total';e={$_.Value.Total}},#{n='Received';e={$_.Value.Received}},#{n='Sent';e={$_.Value.Sent}} |
Sort Total -Descending

Declare a here-string containing variables outside of a loop

I would like to declare a here-string outside of a loop use that string in the loop where the variables get resolved.
My ideal scenario would look like below. This doesn't work as Powershell evaluates the string one time before entering the loop instead of each time inside the loop kind of obvious but bitten by it nevertheless.
$number = "Number $($_)"
1..2 | % { $number }
I know I can use one of these solutions
1..2 | % { "Number $($_)" }
$number = "Number {0}"
1..2 | % { $number -f $_ }
$number = "Number <replace>"
1..2 | % { $number -replace "<replace>", "$_" }
but they have drawbacks I'd like to avoid
Due to the size of the string, declaring it inside the loop obfuscates the logic of the loop making the code less readable.
The formatting solution is too easy to get wrong when many variables are involved.
In the replace solution it's easier to match what get's replaced by what variable but I would have to chain many replace commands.
Edit
Rereading my own question makes it obvious that the actual use case is missing from the question.
Note that ultimately I ended up choosing the formatting option
Following would declare the template with some variables that need replacing in a loop
$sqltemplate = #"
SELECT aud.dpt_mov_hex||aud.dpt_ref||aud.can_typ||TO_CHAR(aud.dte_aud-1,'YYYYMMDD')||'000001' transaction_id,
acc.dos_nbr contract_id, acc.pay_acc_nbr account_id,
CASE WHEN NULL IS NULL THEN unt.nam_unt ELSE unt.nam_unt||'<'||NULL ||'>' END product_id,
aud.dpt_ref, aud.dpt_mov_hex, aud.dpt_mov_dte uitwerkingsdatum,
CASE WHEN can_typ = 0 THEN 'VZ'||aud.dpt_mov_ven_typ ELSE 'VZ'||aud.dpt_mov_ven_typ||'-CR' END transactietype,
aud.dpt_mov_amt_eur bedrag_in_eur, aud.dte_cnv, aud.dpt_mov_fix_eur, aud.dpt_mov_con_inc, aud.dpt_mov_amt_sgn bedrag_teken,
aud.dpt_mov_amt_unt bedrag_in_units, aud.dpt_mov_amt_rte, aud.dpt_mov_amt_val_pre, aud.dpt_mov_amt_val_aft,
aud.dpt_mov_amt_ioc, aud.dte_exe verwerkingsdatum, aud.exe_mng, aud.cmt, aud.trn_nbr, aud.dte_aud datum_aanlevering, aud.can_typ
FROM lfe_dpt_mov_aud aud, vnv_isr_pay_acc acc, vnv_bel_unt unt
WHERE aud.dte_aud >= TO_DATE('$((Get-Date).ToString('dd.MM.yyyy'))', 'DD.MM.YYYY')
AND aud.dpt_ref = '{0}'
AND acc.pay_acc_nbr = '{1}'
AND unt.inv_unt = '{2}'
UNION
SELECT aud.dpt_mov_hex||aud.dpt_ref||aud.can_typ||TO_CHAR(aud.dte_aud-1,'YYYYMMDD')||'000001' transaction_id,
acc.dos_nbr contract_id, acc.pay_acc_nbr account_id,
CASE WHEN itr_rte IS NULL THEN unt.nam_unt ELSE unt.nam_unt||'<'||itr_rte ||'>' END product_id,
aud.dpt_ref, aud.dpt_mov_hex, aud.dpt_mov_dte uitwerkingsdatum,
CASE WHEN can_typ = 0 THEN 'VZ'||aud.dpt_mov_ven_typ ELSE 'VZ'||aud.dpt_mov_ven_typ||'-CR' END transactietype,
aud.dpt_mov_amt_eur bedrag_in_eur, aud.dte_cnv, aud.dpt_mov_fix_eur, aud.dpt_mov_con_inc, aud.dpt_mov_amt_sgn bedrag_teken,
aud.dpt_mov_amt_unt bedrag_in_units, aud.dpt_mov_amt_rte, aud.dpt_mov_amt_val_pre, aud.dpt_mov_amt_val_aft,
aud.dpt_mov_amt_ioc, aud.dte_exe verwerkingsdatum, aud.exe_mng, aud.cmt, aud.trn_nbr, aud.dte_aud datum_aanlevering, aud.can_typ
FROM lfe_dpt_mov_aud aud, vnv_dpt dpt, vnv_isr_pay_acc acc, vnv_bel_unt unt
WHERE aud.dpt_ref = dpt.dpt_ref
AND dpt.pay_acc = acc.pay_acc_nbr
AND dpt.inv_unt = unt.inv_unt
AND aud.dte_aud >= TO_DATE('$((Get-Date).ToString('dd.MM.yyyy'))', 'DD.MM.YYYY')
AND acc.pay_acc_nbr = '{1}'
AND unt.inv_unt = '{2}'
UNION
"#
and this template would get used in a statement such as this
$rolledbackMatchs is an array of custom object containing the three properties: dtp_ref, pay_acc_nbr and inv_unt.
$rolledbackMatches | ForEach-Object { $sqltemplate -f $_.dpt_ref, $_.pay_acc_nbr, $_.inv_unt }
Couple of approaches come to mind:
dot source here-string assignment from a separate file:
# loop.variables.ps1
$myVar = #"
Stuff going on with $_ in here
"#
and then in the loop itself:
1..2 | % { . .\loop.variables.ps1; <# do stuff with $myVar here #> }
Manually invoke string expansion:
$hereString = #'
Stuff (not yet) going on with $_ in here
'#
1..2 | % { $myVar = $ExecutionContext.InvokeCommand.ExpandString($hereString) }
Wrap it in a scriptblock
(as suggested by PetSerAl)
$stringBlock = {
#"
Stuff going on with $_ in here
"#
}
1..2 | % { $myVar = &$stringBlock}
I'm struggling to understand what you're trying to achieve here.
For a start you never define a here-string you just define $number as a string
A here-string would look like this
$number = #"
Number 4
"#
if all you're trying to do is push a number into a string try this
foreach ($number in (1..3)){
"Number $number"
}
which is close to your desired option and less ambiguous

Powershell hashtable with multiple values and one key

Im looking for a data structure/cmdlet that will allow me to add multiple values to a single key in Powershell.
My data would ideally look like this:
KEY-------------------------- VALUES
HOSTNAME1-------------DATABASE1,DATABASE2,DATABASE3
HOSTNAME2-------------DATABASE1,DATABASE2
etc...
I thought a hashtable would do the trick, but I'm unable to do the following:
$servObjects = #{}
$servObjects.Add("server1", #())
$servObjects.get_item("server1") += "database1"
This yields an empty array when I try:
$servObjects.get_item("server1")
I have also tried to do the following, hoping that powershell would understand what I want:
$servObjects2 = #{}
$servObjects2.add($servername, $databasename)
This will unfortunately yield a duplicate key exception
Thanks for any and all input
You basically want a hash table with values that are arrays. You don't have to use $hashtable.get_item or .add
$myHashTable = #{} # creates hash table
$myHashTable.Entry1 = #() #adds an array
$myHashTable.Entry1 += "element1"
$myHashTable.Entry1 += "element2"
This results in the following output:
$myHashTable
Name Value
---- -----
Entry1 {element1, element2}
$myHashTable.Entry1
element1
element2
If you have your data in an array you can group the array and convert to a hash table:
$ary = #()
$ary = $ary + [PSCustomObject]#{RowNumber = 1; EmployeeId = 1; Value = 1 }
$ary = $ary + [PSCustomObject]#{RowNumber = 2; EmployeeId = 1; Value = 2 }
$ary = $ary + [PSCustomObject]#{RowNumber = 3; EmployeeId = 2; Value = 3 }
$ary = $ary + [PSCustomObject]#{RowNumber = 4; EmployeeId = 2; Value = 4 }
$ary = $ary + [PSCustomObject]#{RowNumber = 5; EmployeeId = 3; Value = 5 }
$ht = $ary | Group-Object -Property EmployeeId -AsHashTable
$ht is then:
Name Value
---- -----
3 {#{RowNumber=5; EmployeeId=3; Value=5}}
2 {#{RowNumber=3; EmployeeId=2; Value=3}, #{RowNumber=4; EmployeeId=2; Value=4}}
1 {#{RowNumber=1; EmployeeId=1; Value=1}, #{RowNumber=2; EmployeeId=1; Value=2}}
In your original example, instead of writing
$servObjects.get_item("server1") += "database1"
you had written
$servObjects.server1 += "database1"
it would have worked.
I'm very new to PowerShell, but I prefer to use
$servObjects.Add("key",#())
over
$servObjects.key = #())
because the .Add will throw a duplicate exception if the key is already present in the hashtable, whereas the assignment will replace an existing entry with a new one. For my purposes, I have found that the implicit replacement is (more often than not) an error, either in my logic, or an anomaly in the input data that needs to be handled.
If you know the value at creation time, it would be clearer this way :
[hashtable]$hash = #{
HOSTNAME1 = #(DATABASE1, DATABASE2, DATABASE3);
HOSTNAME2 = #(DATABASE1, DATABASE2);
}
Which will get you the following :
Name Value
---- -----
HOSTNAME2 {DATABASE1, DATABASE2}
HOSTNAME1 {DATABASE1, DATABASE2, DATABASE3}

PowerShell accessing data from hashtable inside hashtable

I have 5 hash tables:
$Monday = #{RUG = "";NRH1 = "";NRH2 = "";ELM = "";BAGVAGT = ""}
$Tuesday = #{RUG = "";NRH1 = "";NRH2 = "";ELM = "";BAGVAGT = ""}
$Wednesday = #{NRH1 = "";NRH2 = "";ELM = "";BAGVAGT = ""}
$Thursday = #{NRH1 = "";NRH2 = "";ELM = "";BAGVAGT = ""}
$Friday = #{NRH1 = "";NRH2 = "";ELM = ""}
That get filled with data. And I can get data out from these either one at at time with $Monday.RUG or all with $Monday | out-string. No problem there.
I'm going to combine those in another hash table 100 times with different data.
So it will be like this:
$Week = #{
1 = #{mo=$Monday;tu=$Tuesday;we=$Wednesday;th=$Thursday;fr=$Friday;val=$value}
2 = #{mo=$Monday;tu=$Tuesday;we=$Wednesday;th=$Thursday;fr=$Friday;val=$value}
}
And so on, until I have 100 different weeks with different values (value will be a calculated number)
But the question is. How do I access the items in the hashtables inside the $week hashtable?
Is there a direct way like $week.1.mo ? or do you need to use a loop?
You can access it using:
$Week[1].mo

PowerShell HashTable - self referencing during initialization

I have a theoretical problem - how to reference a hash table during its initialization, for example, to compute a member based other already stated members.
Remove-Variable myHashTable -ErrorAction Ignore
$myHashTable =
#{
One = 1
Two= 2
Three = ??? # following expressions do not work
# $This.One + $This.Two or
# $_.One + $_.Two
# $myHashTable.One + $myHashTable.Two
# ????
}
$myHashTable.Three -eq 3 # make this $true
Any ideas how to do it? Is it actually possible?
Edit:
This was my solution:
$myHashTable =
#{
One = 1
Two= 2
}
$myHashTable.Three = $myHashTable.One + $myHashTable.Two
This won't be possible using the object initializer syntax I'm afraid. While it is possible to use variables, you'll have to compute the values before creating the object.
I cannot recommend this, but you can iterate the initializer twice or more:
(0..1) | %{
$a = #{
One = 1
Two = $a.One + 1
}
}
(0..2) | %{
$b = #{
One = 1
Two = $b.One + 1
Three = $b.Two + 1
}
}
Make sure all calculations are idempotent, i.e. do not depend on a number of iterations.
You can also recur to this...
sometimes when the hashtable is very long
and can be defined only in 2 or three recurrences...
works fine:
$AAA = #{
DAT = "C:\MyFolderOfDats"
EXE = "C:\MyFolderOfExes"
}
$AAA += #{
Data = $AAA.DAT + "\#Links"
Scripts = $AAA.EXE + "\#Scripts"
ScriptsX = $AAA.EXE + "\#ScriptsX"
}
Note in the second part we are just adding ( += ) more items to the first part... but now... we can refer the items in first part
of the hashtable