I am trying to convert a timestamp to milliseconds in a script using an external program. The external program outputs the time in the shortest possible format
Example the external program can output any of the following formats:
"1:33.823" #1min:33 seconds.823 - shortest
"11:33.823" #11min:33 seconds.823 - short
"1:11:33.823" #1 hour 11min:33 seconds.823 - long
I need this value converted in milliseconds. so I tried [timestamp] $mystring but it complains about the format in the first case. I thought about parsing the format manually with something like
$format='h\:mm\:ss\.ff'
[timespan]::ParseExact($mystring,$format , $null)
But the problem with this approach is that I have to predict every possible output format of the external program and implement all the cases manually (if shortest then $format=m\:ss\.fff else if short then $format=...
Or I can possibly split and define the string, loop from the back and define the attributes of the TimeSpan object manually.
My question is: are there any standard (good practice) conversion methods for my case provided in the language or are my "quick and dirty" solutions common?
Thanks
If you ensure that your time-span strings have at least 3 :-separated components, you can cast them directly to [timespan] (which, behind the scenes, delegates to [timespan]::Parse($inputString, [cultureinfo]::InvariantCulture))[1]
#(
"1:33.823" #1min:33 seconds.823 - shortest
"11:33.823" #11min:33 seconds.823 - short
"1:11:33.823" #1 hour 11min:33 seconds.823 - long
) | ForEach-Object {
$colonCount = ($_ -replace '[^:]').Length
[timespan] ('0:' * [math]::Max(0, (3 - $colonCount - 1)) + $_)
}
The above transforms the input strings to 0:1:33.823, 0:11:33.823, and 1:11:33.823 before casting, i.e. prepends 0: components as needed.
[1] PowerShell by design uses the invariant culture when possible - see this answer for more information.
Related
I'm trying to get a handle on documentation/evidence around when PowerShell does conversions, and when a user needs to be explicit about what they want. A somewhat related question, here, has a broken link that possibly explained scenarios that the type would be adjusted. There are plenty of instances of similar problems though (namely, comparing a string to an int) - for example when you get a number via Read-Host, you're actually getting a string.
Specifically though I've found that certain mechanisms seem to handle a string representation of the number fine - but I want to understand if that is truly because they're handling it as a number, or if the output is correct in appearance, but wrong under the hood. Here's a specific example though, so the question is about Measure-Object and how it handles calculating the Sum of a string property.
PS > $Example = Import-Csv .\Example.csv
PS > $Example
Procedure_Code : 123456789
CPT_Code : J3490
Modifier :
Service_Date : 02/01/2020
Revenue_Code : 259
Units : -100.00
Amount : 55.00
Procedure_Code : 123456789
CPT_Code : J3490
Modifier :
Service_Date : 02/02/2020
Revenue_Code : 259
Units : 100.00
Amount : 55.00
PS > [Math]::Sign($Example[0].Units)
-1
PS > $Example | Measure-Object Amount -Sum
Count : 2
Sum : 110
Property : Amount
PS > $Example | ForEach-Object { $Sum = $Sum + $_.Amount} ; Write-Host "$($Sum)"
55.0055.00 #Plus sign concatenates strings
PS > $Example | ForEach-Object { $Sum = $Sum + [int]$_.Amount} ; Write-Host "$($Sum)"
110 #Expected result if casting the string into an integer type before adding
So basically it seems like [Math]::Sign() works (even though it doesn't accept strings as far as I can tell), and Measure-Object is smarter about strings than a simple plus sign. But is Measure-Object casting my string into an [int]? If I use an example with decimals, the answer is more precise, so maybe it is a [single]?
What is Measure-Object doing to get the right answer, and when shouldn't I trust it?
Measure-Object is a special case, because it is that cmdlet's specific logic that performs automatic type conversion, as opposed to PowerShell's own, general automatic conversions that happen in the context of using operators such as +, passing arguments to commands, calling .NET methods such as [Math]::Sign(), and using values as conditionals.
As your own answer hints at, Measure-Object calls the same code that PowerShell's general automatic type conversions use; these are discussed below.
Automatic type conversions in PowerShell:
In general, PowerShell always attempts automatic type conversion - and a pitfall to those familiar with C# is that even an explicit cast (e.g. [double] '1.2') may not be honored if the target type is ultimately a different one.[1]
Supported conversions:
In addition to supporting .NET's type conversions, PowerShell implements a few built-in custom conversions and supports user-defined conversions by way of custom conversion classes and attribute classes; also, it implicitly tries to call single-parameter target-type constructors and - for string input - a target type's static ::Parse() method, if present; see this answer for details.
To-number conversion and culture-sensitivity when converting to and from strings:
Typically, PowerShell uses culture-invariant conversion, so that, for instance, only . is recognized as the decimal mark in "number strings" that represent floating-point numbers.
String-to-number conversion, including how specific numeric data types are chosen, is covered in this answer.
To-string conversion (including from numbers) is covered in this answer.
Automatic type conversions by context:
Operands (operator arguments):
Some operators, such as -replace and -match, operate on strings only, in which case the operand(s) are invariably converted to strings (which always succeeds):
42 -replace 2, '!' yields '4!': both 42 and 2 were implicitly converted to strings.
Others, such as - and \, operate on numbers only, in which case the operand(s) are invariably converted to numbers (which may fail):
'10' - '2' yields 8, an [int]
Yet others, such as + and *, can operate on either strings or numbers, and it is the type of the LHS that determines the operation type:
'10' + 2 yields '102', the concatenation of string '10' with the to-string conversion of number 2.
By contrast, 10 + '2' yields 12, the addition of the number 10 and the to-number conversion of string '2'.
See this answer for more information.
Command arguments:
If the target parameter has a specific type (other than [object] or [psobject]), PowerShell will attempt conversion.
Otherwise, for literal arguments, the type is inferred from the literal representation; e.g. 42 is passed as an [int], and '42' as a [string].
Values serving as conditionals, such as in if statements:
A conditional must by definition be a Boolean (type [bool]), and PowerShell automatically converts a value of any type to [bool], using the rules described in the bottom section of this answer.
Method arguments:
Automatic type conversion for .NET-method arguments can be tricky, because multiple overloads may have to be considered, and - if the argument's aren't of the expected type - PowerShell has to find the best overload based on supported type conversions - and it may not be obvious which one is chosen.
Executing [Math]::Sign (calling this as-is, without ()) reveals 8(!) different overloads, for various numeric types.
More insidiously, the introduction of additional .NET-method overloads in future .NET versions can break existing PowerShell code, if a new overload then happens to be the best match in a given invocation.
A high-profile example is the [string] type's Split() method - see the bottom section of this answer.
Therefore, for long-term code stability:
Avoid .NET methods in favor of PowerShell-native commands, if possible.
Otherwise, if type conversion is necessary, use casts (e.g. [Math]::Sign([int] '-42')) to guide method-overload resolution to avoid ambiguity.
[1] E.g., the explicit [double] is quietly converted to an [int] in the following statement: & { param([int] $i) $i } ([double] '1.2'). Also, casts to .NET interfaces generally have no effect in PowerShell - except to guide overload resolution in .NET method calls.
I'm not sure this is the be-all and end-all answer, but it works for what I was curious about. For a lot of 'quick script' scenarios where you might use something like Measure-Object - you'll probably get the correct answers you're looking for, albeit maybe slower than other methods.
Measure-Object specifically seems to use [double] for Sum, Average, and StandardDeviation and will indeed throw an error if one of the string values from a CSV Object can't be converted.
I'm still a little surprised that [Math]::Sign() works at all with strings, but seemingly glad it does
if (_measureAverage || _measureSum || _measureStandardDeviation)
{
double numValue = 0.0;
if (!LanguagePrimitives.TryConvertTo(objValue, out numValue))
{
_nonNumericError = true;
ErrorRecord errorRecord = new(
PSTraceSource.NewInvalidOperationException(MeasureObjectStrings.NonNumericInputObject, objValue),
"NonNumericInputObject",
ErrorCategory.InvalidType,
objValue);
WriteError(errorRecord);
return;
}
AnalyzeNumber(numValue, stat);
}
I'm trying to replicate the functionality of the following Python snippit in PowerShell:
allowed_mac_separators = [':', '-', '.']
for sep in allowed_mac_separators:
if sep in mac_address:
test = codecs.decode(mac_address.replace(sep, ''), 'hex')
b64_mac_address = codecs.encode(test, 'base64')
address = codecs.decode(b64_mac_address, 'utf-8').rstrip()
It takes a MAC address, removes the separators, converts it to hex, and then base64. (I did not write the Python function and have no control over it or how it works.)
For example, the MAC address AA:BB:CC:DD:E2:00 would be converted to AABBCCDDE200, then to b'\xaa\xbb\xcc\xdd\xe2\x00', and finally as output b'qrvM3eIA'. I tried doing something like:
$bytes = 'AABBCCDDE200' | Format-Hex
[System.BitConverter]::ToString($bytes);
but that produces MethodException: Cannot find an overload for "ToString" and the argument count: "1". and I'm not really sure what it's looking for. All the examples I've found utilizing that call only have one argument. This works:
[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes('AABBCCDDE200'))
but obviously doesn't convert it to hex first and thus yields the incorrect result. Any help is appreciated.
# Remove everything except word characters from the string.
# In effect, this removes any punctuation ('-', ':', '.')
$sanitizedHexStr = 'AA:BB:CC:DD:E2:00' -replace '\W'
# Convert all hex-digit pairs in the string to an array of bytes.
$bytes = [byte[]] -split ($sanitizedHexStr -replace '..', '0x$& ')
# Get the Base64 encoding of the byte array.
[System.Convert]::ToBase64String($bytes)
For an explanation of the technique used to create the $bytes array, as well as a simpler PowerShell (Core) 7.1+ / .NET 5+ alternative (in short: [System.Convert]::FromHexString('AABBCCDDE200')), see this answer.
As for what you tried:
Format-Hex does not return an array of bytes (directly), its primary purpose is to visualize the input data in hex format for the human observer.
In general, Format-* cmdlets output objects whose sole purpose is to provide formatting instructions to PowerShell's output-formatting system - see this answer. In short: only ever use Format-* cmdlets to format data for display, never for subsequent programmatic processing.
That said, in the particular case of Format-Hex the output objects, which are of type [Microsoft.PowerShell.Commands.ByteCollection], do contain useful data, and do contain the bytes of the transcoded characters of input strings .Bytes property, as Cpt.Whale points out.
However, $bytes = ($sanitizedHexStr | Format-Hex).Bytes would not work in your case, because you'd effectively get byte values reflecting the ASCII code points of characters such as A (see below) - whereas what you need is the interpretation of these characters as hex digits.
But even in general I suggest not relying on Format-Hex for to-byte-array conversions:
On a philosophical note, as stated, the purpose of Format-* cmdlets is to produce for-display output, not data, and it's worth observing this distinction, this exception notwithstanding - the type of the output object could be considered an implementation detail.
Format-Hex converts strings to bytes based on first applying a fixed character transcoding (e.g., you couldn't get the byte representation of a .NET string as-is, based on UTF-16 code units), and that fixed transcoding differs between Windows PowerShell and PowerShell (Core):
In Windows PowerShell, the .NET string is transcoded to ASCII(!), resulting in the loss of non-ASCII-range characters - they are transcoded to literal ?
In PowerShell (Core), that problem is avoided by transcoding to UTF-8.
The System.BitConverter.ToString failed, because $bytes in your code wasn't itself a byte array ([byte[]]), only its .Bytes property value was (but didn't contain the values of interest).
That said, you're not looking to reconvert bytes to a string, you're looking to convert the bytes directly to Base64-encoding, as shown above.
Thank you in advance...
I have String output:
2021-12-23
the string Base.Type is System.Object
I would like to convert it to:
12-23-2021
or
12/23/2021
any ideas?
Note: This question is a near-duplicate of Change date format from "yyyymmdd" to "mm/dd/yyyy"; while the two are closely related, this question is more overtly focused on both parsing from a string to a date and then back to a string, using a different format; by contrast, the linked question has the from aspect covered as an incidental part of the question.
Bali C, in a comment on the question, came up with the most concise solution, which only requires a small tweak:
# Note the "\"-escaped separators - see below.
PS> Get-Date -Date 2021-12-23 -Format dd\-MM\-yyyy # or dd\/MM\/yyyy
23-12-2021
This will work with most cultures in effect, because format yyyy-MM-dd (as exemplified by your sample input string, '2021-12-23') is a recognized format in all but a few cultures.
PowerShell generally tries to use the invariant culture in order for code to work the same across cultures, but in the case of cmdlet arguments actually is culture-sensitive, which is a historical accident that cannot be fixed without breaking backward compatibility.
See below for a truly culture-invariant solution, which also explains why separator chars. - and / should be escaped for maximum robustness.
Since PowerShell's casts use the invariant culture and format yyyy-MM-dd (as exemplified by your sample input string, '2021-12-23') happens to be a recognized format in that culture, you can simply:
cast your string to [datetime] to create a System.DateTime instance
and then call .ToString() with the desired format string to produce the desired string format, as shown in the many answers to this question.
The following should work irrespective of the specific culture in effect:
PS> ([datetime] '2021-12-23').ToString('MM\-dd\-yyyy')
12-23-2021
PS> ([datetime] '2021-12-23').ToString('MM"/"dd"/"yyyy')
12/23/2021
Note that the separator chars. - and / are escaped / quoted so as to indicate that they're to be used verbatim (you're free to choose between \-escaping and embedded quoting).
This is necessary, because characters such as / are by default interpreted as placeholders for the culture-appropriate date separator, for instance, so - depending on the culture in effect (as reflected in $PSCulture) - they may be translated to a different character, such as ., for instance.
If your original string representation were not recognized in the invariant culture, you can use System.DateTime.ParseExact to parse your input string into a [datetime] (System.DateTime) instance before calling .ToString() as described above:
[datetime]::ParseExact(
'2021-12-23',
'yyyy\-MM\-dd',
$null # current culture
).ToString('MM"-"dd"-"yyyy')
The above again yields '12-23-2021'.
Note that, due to use of $null, the current culture is used for parsing, which has no effect in this case, however, given that the separator chars. use embedded quoting, and given that only digits are involved, not (culture-dependent) names, such as month names.
If you do need the context of a specific culture, pass a [cultureinfo] instance (System.Globalization.CultureInfo) instead of $null; e.g., to parse in the context of the French (France) culture, pass [cultureinfo] 'fr-FR'.
I'm using ffprobe to get the sexagesimal time and after trimming the last three (unnecessary) digits I get the following format:
#examples
0:05:51.15
11:03:15.24
Is there a way to add these 2 so that the result is 11:09:06.39 ?
You can cast the strings to type [timespan], which allows you to perform arithmetic on the resulting objects, and to apply custom formatting on output:
PS> ([timespan] ' 0:05:51.15' + [timespan] '11:03:15.24').ToString('h\:mm\:ss\.ff')
11:09:06.39
Note: If there's a chance that the resulting time span exceeds 24 hours, more work is needed.[1]
Note how \-escaping must be used to specify the output separators as literals.
In this case, the simpler .ToString('g') would have yielded the same output, but only in cultures that use . as the decimal separator, because the standard g format specifier is culture-sensitive.
See the [timespan]::ToString() documentation as well as the documentation on standard and custom time-span format specifiers.
By contrast, PowerShell uses the invariant culture when interpreting the input format cast to [timespan], where . is the decimal separator; similarly, using a [timespan] instance in expandable strings yields a culture-invariant representation; e.g.:
[timespan] '11:03:15.24' always works, irrespective of the current culture, because the invariant culture expects . as the decimal separator.
"$([timespan] '1' - 1)" always yields 23:59:59.9999999, irrespective of the current culture[2].
As No Refunds No Returns notes, if you're dealing with differently formatted input, possibly from a different culture, you can use [timespan]::Parse() / [timespan]::ParseExact() / [timespan]::TryParseExact()
Parsing the standard formats of a given culture:
[timespan]::Parse(' 0:05:51,15', 'fr-FR') # OK
Note the , as the decimal separator.
If you omit the culture argument (or pass $null), the current culture is applied. Note how that differs from using a [timespan] cast, which is always culture-invariant (and assumes . as the decimal separator).
Parsing with a custom format:
[timespan]::ParseExact(' 0:05:51,15'.Trim(), 'h\:mm\:ss\,ff', $null) # OK
Note that using such a literal custom format is never culture-sensitive, because all the separators must be specified as - escaped - literals (e.g., \:), so $null is passed as the culture argument (IFormatProvider).
Conversely, passing a specific culture only makes sense with the culture-sensitive standard format specifiers, g and G.
Parsing with a culture-aware custom format:
If you don't know what culture will be in effect at runtime, but you want to respect that culture's decimal separator in combination with a custom format, you need to dynamically embed the current culture's decimal separator in your custom format string:
$tsText = ' 0:05:51.15'
[timespan]::ParseExact($tsText.Trim(),
('h\:mm\:ss\{0}ff' -f [cultureinfo]::CurrentCulture.NumberFormat.NumberDecimalSeparator),
$null
) # OK in cultures that use "." as the decimal separator
[1] h and hh only ever reflect the hours not included in full days in the input time span. To reflect the number of days too, prepend something like d\. - there is no format specifier that allows you express the total number of hours across multiple days, but you can use general-purpose string formatting to achieve that - do note, however, that you'll also need custom parsing code in order to convert the resulting string back to a [timespan] instance:
$ts = [timespan] ' 1:05:51.15' + [timespan] '23:03:15.24'
'{0}:{1:mm\:ss\.ff}' -f [Math]::Floor($ts.TotalHours), $ts
[2] At the .NET level, calling .ToString() on objects typically yields a culture-sensitive representation (if the type supports it), but with [timespan] the output happens to be culture-invariant. By contrast, PowerShell explicitly uses the invariant culture behind the scenes in casts and string interpolation, so that "$var" and $var.ToString() may not yield the same representation - see this answer for details.
Problem
I am working on a script that has a user provide a specific IP address and I want to mask this IP in some fashion so that it isn't stored in the logs. My problem is, that I can easily do this when I know what the first three values of the IP typically are; however, I want to avoid storing/hard coding those values into the code to if at all possible. I also want to be able to replace the values even if the first three are unknown to me.
Examples:
10.11.12.50 would display as XX.XX.XX.50
10.12.11.23 would also display as XX.XX.XX.23
I have looked up partial string replacements, but none of the questions or problems that I found came close to doing this. I have tried doing things like:
# This ended up replacing all of the numbers
$tempString = $str -replace '[0-9]', 'X'
I know that I am partway there, but I aiming to only replace only the first 3 sets of digits so, basically every digit that is before a '.', but I haven't been able to achieve this.
Question
Is what I'm trying to do possible to achieve with PowerShell? Is there a best practice way of achieving this?
Here's an example of how you can accomplish this:
Get-Content 'File.txt' |
ForEach-Object { $_ = $_ -replace '\d{1,3}\.\d{1,3}\.\d{1,3}','xx.xx.xx' }
This example matches a digit 1-3 times, a literal period, and continues that pattern so it'll capture anything from 0-999.0-999.0-999 and replace with xx.xx.xx
TheIncorrigible1's helpful answer is an exact way of solving the problem (replacement only happens if 3 consecutive .-separated groups of 1-3 digits are matched.)
A looser, but shorter solution that replaces everything but the last .-prefixed digit group:
PS> '10.11.12.50' -replace '.+(?=\.\d+$)', 'XX.XX.XX'
XX.XX.XX.50
(?=\.\d+$) is a (positive) lookahead assertion ((?=...)) that matches the enclosed subexpression (a literal . followed by 1 or more digits (\d) at the end of the string ($)), but doesn't capture it as part of the overall match.
The net effect is that only what .+ captured - everything before the lookahead assertion's match - is replaced with 'XX.XX.XX'.
Applied to the above example input string, 10.11.12.50:
(?=\.\d+$) matches the .-prefixed digit group at the end, .50.
.+ matches everything before .50, which is 10.11.12.
Since the (?=...) part isn't captured, it is therefore not included in what is replaced, so it is only substring 10.11.12 that is replaced, namely with XX.XX.XX, yielding XX.XX.XX.50 as a result.