PowerShell scripts sometimes pipe to %{0}:
[byte[]]$bytes = 0..65535 | %{0};
I know % is an alias for ForEach-Object and $_ represents the current pipeline object.
Is this solely to avoid output? Isn’t there a smarter way of doing it?
The pipeline on the right-hand side simply outputs the integer 0 65536 times - which when cast to byte[] produces a byte array of length 65536 with all values initialized to 0.
You could also have done:
[byte[]]$bytes = ,0 * 65536
As PetSerAl hints at, this is unnecessary since arrays of numerical value types initialize all items to 0 anyways, meaning that simply creating a new array, like so:
# using the new constructor keyword, PowerShell version > 5,
[byte[]]$bytes = [byte[]]::new(65536)
# using New-Object, PowerShell version > 2
[byte[]]$bytes = New-Object 'byte[]' 65536
would also have given you the exact same result
Related
Can an array be used as the key in a hashtable? How can I reference the hashtable item with an array key?
PS C:\> $h = #{}
PS C:\> $h[#(1,2)] = 'a'
PS C:\> $h
Name Value
---- -----
{1, 2} a # looks like the key is a hash
PS C:\> $h[#(1,2)] # no hash entry
PS C:\> $h.Keys #
1
2
PS C:\> $h[#(1,2)] -eq 'a'
PS C:\> $h[#(1,2)] -eq 'b'
PS C:\> foreach ($key in $h.Keys) { $key.GetType() } # this is promising
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
PS C:\> $PSVersionTable.PSVersion.ToString()
7.1.4
While you can use arrays as hashtable keys, doing so is impractical:
Update: There is a way to make arrays work as hashtable keys, but it requires nontrivial effort during construction of the hashtable - see this answer.
You'll have the use the very same array instances as both the keys and for later lookups.
The reason is that arrays, which are instances of .NET reference types (as opposed to value types such as integers), use the default implementation of the .GetHashCode() method to return a hash code (as used in hashtables), and this default implementation returns a different code for each instance - even for two array instances that one would intuitively think of as "the same".
In other words: you'll run into the same problem trying to use instances of any such .NET reference type as hashtable keys, including other collection types - unless a given type happens to have a custom .GetHashCode() implementation that explicitly considers distinct instances equal based on their content.
Additionally, it makes use of PowerShell's indexer syntax ([...]) awkward, because the array instance must be nested, with the unary form of ,, the array constructor operator. However, dot notation (property access) works as usual.
$h = #{}
# The array-valued key.
$key = 1, 2
$h[$key] = 'a'
# IMPORTANT:
# The following lookups work, but only because
# the *very same array instance* is used for the lookup.
# Nesting required so that PowerShell doesn't think that
# *multiple* keys are being looked up.
$h[, $key]
# Dot notation works normally.
$h.$key
# Does NOT work, because a *different array instance* is used.
$h.#(1,2)
A simple test for whether a given expression results in the same hashtable lookup every time and is therefore suitable as a key is to call the .GetHashCode() method on it repeatedly; only if the same number is returned every time (in a given session) can the expression be used:
# Returns *different* numbers.
#(1, 2).GetHashCode()
#(1, 2).GetHashCode()
To inspect a given object or type for whether it is (an instance of) a .NET reference type vs. value type:
# $false is returned in both cases, confirming that the .NET array
# type is a *reference type*
#(1, 2).GetType().IsValueType
[Array].IsValueType
Workaround:
A workaround would be to use string representations of arrays, though coming up with unique (enough) ones may be a challenge.
In the simplest case, use PowerShell's string interpolation, which represents arrays as a space-separated list of the elements' (stringified) values; e.g. "$(1, 2)" yields verbatim 1 2:
$h = #{}
# The array to base the key on.
$array = 1, 2
# Use the *stringified* version as the key.
$h["$array"] = 'a'
# Works, because even different array instances with equal-valued
# instances of .NET primitive types stringify the same.
# '1 2'
$h["$(1, 2)"]
iRon points out that this simplistic approach can lead to ambiguity (e.g., a single '1 2' string would result in the same key as array 1, 2) and recommends the following instead:
a more advanced/explicit way for array keys would be:
joining their elements with a non-printable character; e.g.
$key = $array -join [char]27
or, for complex object array elements, serializing the array:
$key = [System.Management.Automation.PSSerializer]::Serialize($array)
Note that even the XML (string)-based serialization provided by the System.Management.Automation.PSSerializer class (used in PowerShell remoting and background jobs for cross-process marshaling) has its limits with respect to reliably distinguishing instances, because its recursion depth is limited - see this answer for more information; you can increase the depth on demand, but doing so can result in very large string representations.
A concrete example:
using namespace System.Management.Automation
$ht = #{}
# Use serialization on an array-valued key.
$ht[[PSSerializer]::Serialize(#(1, 2))] = 'a'
# Despite using a different array instance, this
# lookup succeeds, because the serialized representation is the same.
$ht[[PSSerializer]::Serialize(#(1, 2))] # -> 'a'
The primary cause of your problems here is that PowerShell's index access operator [] supports multi-index access by enumerating any array values passed.
To understand why, let's have a look at how the index accessor [...] actually works in PowerShell. Let's start with a simple hashtable, with 2 entries using scalar keys:
$ht = #{}
$ht['a'] = 'This is value A'
$ht['b'] = 'This is value B'
Now, let's inspect how it behaves!
Passing a scalar argument resolves to the value associated with the key represented by said argument, so far so good:
PS ~> $ht['a']
This is value A
But we can also pass an array argument, and all of a sudden PowerShell will try to resolve all items as individual keys:
PS ~> $ht[#('a', 'b')]
This is value A
This is value B
PS ~> $ht[#('b', 'a')] # let's try in reverse!
This is value B
This is value A
Now, to understand what happens in your example, let's try an add an entry with an array reference as the key, along with two other entries where the key is the individual values fround in the array:
$ht = #{}
$keys = 1,2
$ht[$keys[0]] = 'Value 1'
$ht[$keys[1]] = 'Value 2'
$ht[$keys] = 'Value 1,2'
And when we subsequently try to resolve the last entry using our array reference:
PS ~> $ht[$keys]
Value 1
Value 2
Oops! PowerShell unraveled the $keys array, and never actually attempted to resolve the entry associated with the key corresponding to the array reference in $keys.
In other words: The index accessor cannot be used to resolve dictionary entries by key is the key type is enumerable
So, how does one access an entry by array reference without having PowerShell unravel the array?
Use the IList.Item() parameterized property instead:
PS ~> $ht.Item($keys)
Value 1,2
Thought I have read enough examples here and elsewhere. Still I fail creating arrays in Power Shell.
With that code I hoped to create slices of pair values from an array.
$values = #('hello','world','bonjour','moon','ola','mars')
function slice_array {
param (
[String[]]$Items
)
[int16] $size = 2
$pair = [string[]]::new($size) # size is 2
$returns = [System.Collections.ArrayList]#()
[int16] $_i = 0
foreach($item in $Items){
$pair[$_i] = $Item
$_i++;
if($_i -gt $size - 1){
$_i = 0
[void]$returns.Add($pair)
}
}
return $returns
}
slice_array($values)
the output is
ola
mars
ola
mars
ola
mars
I would hope for
'hello','world'
'bonjour','moon'
'ola','mars'
Is possible to slice that array to an array of arrays with length 2 ?
Any explenation why it doesn't work as expected ?
How should the code be changed ?
Thanks for any hint to properly understand Arrays in PowerShell !
Here's a PowerShell-idiomatic solution (the fix required for your code is in the bottom section):
The function is named Get-Slices to adhere to PowerShell's verb-noun naming convention (see the docs for more information).
Note: Often, the singular form of the noun is used, e.g. Get-Item rather than Get-Items, given that you situationally may get one or multiple output values; however, since the express purpose here is to slice a single object into multiple parts, I've chosen the plural.
The slice size (count of elements per slice) is passed as a parameter.
The function uses .., the range operator, to extract a single slice from an array.
It uses PowerShell's implicit output behavior (no need for return, no need to build up a list of return values explicitly; see this answer for more information).
It shows how to output an array as a whole from a function, which requires wrapping it in an auxiliary single-element array using the unary form of ,, the array constructor operator. Without this auxiliary array, the array's elements would be output individually to the pipeline (which is also used for function / script output; see this answer for more information.
# Note: For brevity, argument validation, pipeline support, error handling, ...
# have been omitted.
function Get-Slices {
param (
[String[]] $Items
,
[int] $Size # The slice size (element count)
)
$sliceCount = [Math]::Ceiling($Items.Count / $Size)
if ($sliceCount -le 1) {
# array is empty or as large as or smaller than a slice? ->
# wrap it *twice* to ensure that the output is *always* an
# *array of arrays*, in this case containing just *one* element
# containing the original array.
,, $Items
}
else {
foreach ($offset in 0..($sliceCount-1)) {
, $Items[($offset * $Size)..(($offset+1) * $Size - 1)] # output this slice
}
}
}
To slice an array into pairs and collect the output in an array of arrays (jagged array):
$arrayOfPairs =
Get-Slices -Items 'hello','world','bonjour','moon','ola','mars' -Size 2
Note:
Shell-like syntax is required when you call functions (commands in general) in PowerShell: arguments are whitespace-separated and not enclosed in (...) (see this answer for more information)
Since a function's declared parameters are positional by default, naming the arguments as I've done above (-Item ..., -Size ...) isn't strictly necessary, but helps readability.
Two sample calls:
"`n-- Get pairs (slice count 2):"
Get-Slices -Items 'hello','world','bonjour','moon','ola','mars' -Size 2 |
ForEach-Object { $_ -join ', ' }
"`n-- Get slices of 3:"
Get-Slices -Items 'hello','world','bonjour','moon','ola','mars' -Size 3 |
ForEach-Object { $_ -join ', ' }
The above yields:
-- Get pairs (slice count 2):
hello, world
bonjour, moon
ola, mars
-- Get slices of 3:
hello, world, bonjour
moon, ola, mars
As for what you tried:
The only problem with your code was that you kept reusing the very same auxiliary array for collecting a pair of elements, so that subsequent iterations replaced the elements of the previous ones, so that, in the end, your array list contained multiple references to the same pair array, reflecting the last pair only.
This behavior occurs, because arrays are instance of reference types rather than value types - see this answer for background information.
The simplest solution is to add a (shallow) clone of your $pair array to your list, which ensures that each list entry is a distinct array:
[void]$returns.Add($pair.Clone())
Why you got 3 equal pairs instead of different pairs:
.Net (powershell based on it) is object-oriented language and it has consept of reference types and value types. Almost all types are reference types.
What happens in your code:
You create $pair = [string[]] object. $pair variable actually stores memory address of (reference to) [string[]] object, because arrays are reference types
You fill $pair array with values
You add (!) $pair to $returns. Remember that $pair is reference to memory block. And when you add it to $returns, it adds memory address of [string[]] you wrote values to.
You repeat step2: You fill $pair array with different values, but address of this array in memory keeps the same. Doing this you actually replace values from step2 with new values in the same $pair object.
= // = step3
= // = step4
= // = step3
As a result: in $returns there are three same memory addresses: [[reference to $pair], [reference to $pair], [reference to $pair]]. And $pair values were overwritten by code with last pair values.
On output it works like this:
Powershell looks at $results which is array.
Powershell looks to $results[0] which reference to $pair
Powershell outputs reference to $pair[0]
Powershell outputs reference to $pair[1]
Powershell looks to $results[1] which reference to $pair
Powershell outputs reference to $pair[0]
Powershell outputs reference to $pair[1]
Powershell looks to $results[1] which reference to $pair
Powershell outputs reference to $pair[0]
Powershell outputs reference to $pair[1]
So you see, you triple output the object from the same memory address. You overwritten it 3 times in slice_array and now it stores only last pair values.
To fix it in your code, you should create a new $pair in memory: add $pair = [string[]]::new($size) just after $returns.Add($pair)
I have a "structured" file (logical fixed-length records) from a legacy program on a legacy (non-MS) operating system. I know how the records were structured in the original program, but the original O/S handled structured data as a sequence of bytes for file I/O, so a hex dump won't show you anything more than what the record length is (there are marker bytes and other record overhead imposed by the access method API used to generate the file originally).
Once I have the sequence of bytes in a Powershell variable, with the overhead bytes "cut away", how can I convert this into a structured object? Some of the "fields" are 16-bit integers, some are strings of the form [s]data (where [s] is a byte giving the length of the "real" data in that field), some are BCD coded fixed-point numbers, some are IEEE floats.
(I haven't been specific about the structure, either on the Powershell side or on the legacy side, because I am seeking a more-or-less 'generic' solution/technique, as I actually have several different files with different record structures to process.)
Initially, I tried to do it by creating a type that could take the buffer and overwrite a struct so that all the fields were nicely filled in. However, certain issues arose (regarding struct layout, fixed buffers and mixing fixed and managed members) and I also realised that there was no guarantee that the data in the buffer would be properly (or even legally) aligned. Decided to try a more programmatic path.
"Manual" parsing is out, so how about automatic parsing? You're going to need to define the members of your PSobject at some point, why not do it in a way that can help programmatically parse the data. This method does not require the data in the buffer to be correctly aligned or even contiguous. You can also have fields overlap to separate raw unions into the individual members (though, typically, only one will contain a "correct" value).
First step, build a hash table to identify the members, the offset in the buffer, their data types and, if an array, the number of elements :
$struct = #{
field1 = 0,[int],0; # 0 means not an array
field2 = 4,[byte],16; # a C string maybe
field3 = 24,[char],32; # wchar_t[32] ? note: skipped over bytes 20-23
field4 = 56,[double],0
}
# the names field1/2/3/4 are arbitrary, any valid member name may be used (but not
# necessarily any valid hash key if you want a PSObject as the end result).
# also, the values could be hash tables instead of arrays. that would allow
# descriptive names for the values but doesn't affect the end result.
Next, use [BitConverter] to extract the required data. The problem here is that we need to call the correct method for all the varying types. Just use a (big) switch statement. The basic principle is the same for most values, get the type indicator and initial offset from the $struct definition then call the correct [BitConverter] method and supply the buffer and initial offset, update the offset to where the next element of an array would be and then repeat for as many array elements as are required. The only trap here is that the data in the buffer must have the same format as expected by [BitConverter], so for the [double] example, the bytes in the buffer must conform to IEEE-754 floating point format (assuming that [BitConverter]::ToDouble() is used). Thus, for example, raw data from a Paradox database will need some tweeking because it flips the high bit to simplify sorting.
$struct.keys | foreach {
# key order is undefined but that won't affect the final object's members
$hashobject = #{}
} {
$fieldoffs = $struct[$_][0]
$fieldtype = $struct[$_][1]
if (($arraysize = $struct[$_][2]) -ne 0) { # yes, I'm a C programmer from way back
$array = #()
} else {
$array = $null
}
:w while ($arraysize-- -ge 0) {
switch($fieldtype) {
([int]) {
$value = [bitconverter]::toint32($buffer, $fieldoffs)
$fieldoffs += 4
}
([byte]) {
$value = $buffer[$fieldoffs++]
}
([char]) {
$value = [bitconverter]::tochar($buffer, $fieldoffs)
$fieldoffs += 2
}
([string]) { # ANSI string, 1 byte per character
$array = new-object string (,[char[]]$buffer[$fieldoffs..($fieldoffs+$arraysize)])
# $arraysize has already been decremented so don't need to subtract 1
break w # "array size" was actually string length so don't loop
#
# description:
# first, get a slice of the buffer as a byte[] (assume single byte characters)
# next, convert each byte to a char in a char[]
# then, invoke the constructor String(Char[])
# finally, put the String into $array ready for insertion into $hashobject
#
# Note the convoluted syntax - New-Object expects the second argument to be
# an array of the constructor parameters but String(Char[]) requires only
# one argument that is itself an array. By itself,
# [char[]]$buffer[$fieldoffs..($fieldoffs+$arraysize)]
# is treated by PowerShell as an argument list of individual chars, corrupting the
# constructor call. The normal trick is to prepend a single comma to create an array
# of one element which is itself an array
# ,[char[]]$buffer[$fieldoffs..($fieldoffs+$arraysize)]
# but this won't work because of the way PowerShell parses the command line. The
# space before the comma is ignored so that instead of getting 2 arguments (a string
# "String" and the array of an array of char), there is only one argument, an array
# of 2 elements ("String" and array of array of char) thereby totally confusing
# New-Object. To make it work you need to ALSO isolate the single element array into
# its own expression. Hence the parentheses
# (,[char[]]$buffer[$fieldoffs..($fieldoffs+$arraysize)])
#
}
}
if ($null -ne $array) {
# must be in this order* to stop the -ne from enumerating $array to compare against
# $null. this would result in the condition being considered false if $array were
# empty ( (#() -ne $null) -> $null -> $false ) or contained only one element with
# the value 0 ( (#(0) -ne $null) -> (scalar) 0 -> $false ).
$array += $value
# $array is not $null so must be an array to which $value is appended
} else {
# $array is $null only if $arraysize -eq 0 before the loop (and is now -1)
$array = $value
# so the loop won't repeat thus leaving this one scalar in $array
}
}
$hashobject[$_] = $array
}
#*could have reversed it as
# if ($array -eq $null) { scalar } else { collect array }
# since the condition will only be true if $array is actually $null or contains at
# least 2 $null elements (but no valid conversion will produce $null)
At this point there is a hash table, $hashobject, with keys equal to the field names and values containing the bytes from the buffer arranged into single (or arrays of) numeric (inc. char/boolean) values or (ANSI) strings. To create a (proper) object, just invoke New-Object -TypeName PSObject -Property $hashobject or use [PSCustomObject]$hashobject.
Of course, if the buffer actually contained structured data then the process would be more complicated but the basic procedure would be the same. Note also that the "types" used in the $struct hash table have no direct effect on the resultant types of the object members, they are only convenient selectors for the switch statement. It would work just as well with strings or numbers. In fact, the parentheses around the case labels are because switch parses them the same as command arguments. Without the parentheses, the labels would be treated as literal strings. With them, the labels are evaluated as a type object. Both the label and the switch value are then converted to strings (that's what switch does for values other than script blocks or $null) but each type has a distinct string representation so the case labels will still match up correctly. (Not really on point but still interesting, I think.)
Several optimisations are possible but increase the complexity slightly. E.g.
([byte]) { # already have a byte[] so why collect bytes one at a time
if ($arraysize -ge 0) { # was originally -gt 0 so want a byte[]
$array = [byte[]]$buffer[$fieldoffs..($fieldoffs+$arraysize)]
# slicing the byte array produces an object array (of bytes) so cast it back
} else { # $arraysize was 0 so just a single byte
$array = $buffer[$fieldoffs]
}
break w # $array ready for insertion into $hashobject, don't need to loop
}
But what if my strings are actually Unicode?, you say. Easy, just use existing methods from the [Text.Encoding] class,
[string] { # Unicode string, 2 (LE) bytes per character
$array = [text.encoding]::unicode.getstring([byte[]]$buffer[$fieldoffs..($fieldoffs+$arraysize*2+1)])
# $arraysize should be the string length so, initially, $arraysize*2 is the byte
# count and $arraysize*2-1 is the end index (relative to $fieldoffs) but $arraysize
# was decremented so the end index is now $arraysize*2+1, i.e. length*2-1 = (length-1)*2+1
break w # got $array, no loop
}
You could also have both ANSI and Unicode by utilising a different type indicator for the ANSI string, maybe [char[]]. Remember, the type indicators do not affect the result, they just have to be distinct (and hopefully meaningful) identifiers.
I realise that this is not quite the "just dump the bytes into a union or variant record" solution mentioned in the OPs comment but PowerShell is based in .NET and uses managed objects where this sort of thing is largely prohibited (or difficult to get working, as I found). For example, assuming you could just dump raw chars (not bytes) into a String, how would the Length property get updated? This method also allows some useful preprocessing such as splitting up unions as noted above or converting raw byte or char arrays into the Strings they represent.
I often see the usage of # symbol in Powershell coding. Want to ask what is it used for? Example as below
$DistributionPointGroups = #("London")
#() is the array operator, which makes sure that even a single item (or zero) is returned as an array.
THE ARRAY SUB-EXPRESSION OPERATOR
The array sub-expression operator creates an array, even if it
contains zero or one object.
The syntax of the array operator is as follows:
#( ... )
You can use the array operator to create an array of zero or
one object.
PS C:\>$a = #("One")
PS C:\>$a.Count
1
PS C:\>$b = #()
PS C:\>$b.Count
0
Source: about_Arrays
What's the best way to round down to nearest whole number in PowerShell?
I am trying [math]::truncate but its not giving me predictable results.
Example:
$bla = 17.2/0.1
[math]::truncate($bla)
outputs 171 instead of the expected 172!
$bla = 172
[math]::truncate($bla)
outputs 172
I just need something that works.... and must always round down (i.e round($myNum + 0.5) won't work due to baker's rounding which may round up if the number has a 0.5 component).
Ah, I see. Looks like the datatype needs to be decimal:
[decimal] $de = 17.2/.1
[double] $db = 17.2/.1
[math]::floor($de)
172
[math]::floor($db)
171
http://msdn.microsoft.com/en-us/library/system.math.floor(v=vs.85).aspx
The Math::Floor function combined with [decimal] declaration should give you the results you want.
[Math]::Floor([decimal](17.27975/0.1))
returns = 172
The issue you are encountering with the original 17.2/0.1 division example is due to inaccuracy in the floating-point representation of the given decimal values (as mentioned in Joey's comment on another answer). You can see this in PowerShell by examining the round-trip representation of the final value:
PS> $bla = 17.2/0.1
PS> $bla.GetType().FullName
System.Double
PS> $bla.ToString()
172
PS> $bla.ToString('r')
171.99999999999997
A simple way to get around this is to declare the result as int, as PowerShell will automatically round to the the result to the nearest integer value:
PS> [int]$bli = 17.2/0.1
PS> $bli.GetType().FullName
System.Int32
PS> $bli.ToString()
172
Note that this uses the default .NET method of MidpointRounding.ToEven (also known as banker's rounding). This has nice statistical properties when tabulating large numbers of numeric values, but can also be changed to the simpler away-from-zero method:
function round( $value, [MidpointRounding]$mode = 'AwayFromZero' ) {
[Math]::Round( $value, $mode )
}
PS> [int]3.5
4
PS> [int]4.5
4
PS> round 3.5
4
PS> round 4.5
5
Another option is to use a more accurate representation for the original values, which will avoid the issue entirely:
PS> $bld = [decimal]17.2/0.1
PS> $bld.GetType().FullName
System.Decimal
PS> $bld.ToString()
172
[Math]::floor($x) is the built-in way to do it.
Just be aware of how it will behave with negative numbers. [Math]::floor(5.5) returns 5, but [Math]::floor(-5.5) returns -6.
If you need the function to return the value closest to zero, you'll need:
If ($x -ge 0) {
[Math]::Floor($x)
} Else {
[Math]::Ceiling($x)
}