Why is PS Get-ChildItem so difficult - powershell

I did a ton of reading and searching about a way to have Get-ChildItem return a dir listing in wide format, in alphabetical order, with the number of files and directories in the current directory. Here is a image of what I ended up with, but not using GCI.
I ended up writing a small PS file.
$bArgs = "--%/c"
$cArgs = "Dir /n/w"
& cmd.exe -ArgumentList $bArgs $cArgs
As you can see I ended up using the old cmd.exe and passing the variables I wanted. I made an alias in my PS $Profile to call this script.
Can this not be accomplished in PS v5.1? Thanks for any help or advice for an old noob.

PowerShell's for-display formatting differs from cmd.exe's, so if you want the formatting of the latter's internal dir command, you'll indeed have to call it via cmd /c, via a function you can place in your $PROFILE file (note that aliases in PowerShell are merely alternative names and can therefore not include baked-in arguments):
function lss { cmd /c dir /n /w /c $args }
Note that you lose a key benefit of PowerShell: the ability to process rich objects:
PowerShell-native commands output rich objects that enable robust programmatic processing; e.g., Get-ChildItem outputs System.IO.FileInfo and System.IO.DirectoryInfo instances; the aspect of for-display formatting is decoupled from the data output, and for-display formatting only kicks in when printing to the display (host), or when explicitly requested.
For instance, (Get-ChildItem -File).Name returns an array of all file names in the current directory.
By contrast, PowerShell can only use text to communicate with external programs, which makes processing cumbersome and brittle, if information must be extracted via text parsing.
As Pierre-Alain Vigeant notes, the following PowerShell command gives you at least similar output formatting as your dir command, though it lacks the combined-size and bytes-free summary at the bottom:
Get-ChildItem | Format-Wide -AutoSize
To wrap that up in a function, use:
function lss { Get-ChildItem #args | Format-Wide -Autosize }
Note, however, that - due to use of a Format-* cmdlet, all of which output objects that are formatting instructions rather than data - this function's output is also not suited to further programmatic processing.
A proper solution would require you to author custom formatting data and associate them with the System.IO.FileInfo and System.IO.DirectoryInfo types, which is nontrivial however.
See the conceptual about_Format.ps1xml help topic, Export-FormatData, Update-FormatData, and this answer for a simple example.

Related

Running help <command> and piping output to Where-Object or Select-Object returns empty rows

Running the command help firewall | Select-Object Category. The result is one column blank Category.
The strange thing is that the empty rows number represent the amount of rows that help firewall would result without calling piping it to Select-Object
Or I'm trying to filter the output of help firewall to return only rows with Name that starts with "Get". Running help firewall | Where-Object Name -like "Get" just returns nothing.
Why aren't these pipes on help working? They are working perfectly on other commands.
Powershell Version 5.1 and using default windows console.
To complement Zilog80's helpful answer with background information:
Get-Command help reveals that help is not a mere alias of the Get-Help cmdlet, but a (built-in) function (submit $function:help to see its definition).
As you've noticed yourself:
while Get-Help outputs an object ([pscsustomobject]) with properties that reflect help-topic metadata such as Category, which is then rendered as display text by PowerShell's output-formatting system,
the help function returns strings - a stream of text lines representing the rendered help topic - of necessity.
You can observe the difference in output type by piping to the Get-Member cmdlet (help firewall | Get-Member vs. Get-Help firewall | Get-Member)
The purpose of the help function is to wrap Get-Help with interactive paging, to allow convenient navigation through lengthy help topics that don't fit onto a single console (terminal) screen.
This paging is provided via an external program (by default, more.com on Windows, and less on Unix-like platforms, configurable via $env:PAGER, but only in PowerShell (Core) 7+), and since PowerShell only "speaks text" when communicating with external programs, help must send a stream of strings (lines for display) to them, which it does via
Out-String -Stream.
Note:
When the external paging programs find that their stdout stream isn't connected to a console (terminal), they take no action other than simply passing the input through (in Unix terms, they then behave like cat).
Hypothetically, the help function itself could determine this condition and then (a) not pipe to the paging program and (b) relay the object output by Get-Help as-is.[1] However, determining a command's output target from inside that command, using PowerShell code, may not even be possible.
[1] The function actually already takes this action when a custom pager defined via $env:PAGER is found to be a PowerShell command rather than an external program.
Check the feedback from help help in PowerShell :
You can also type `help` or `man`, which displays one screen of text at a
time. Or, ` -?`, that is identical to `Get-Help`, but only
works for cmdlets.
The helpcommand display "screen of text" which means it is outputting [System.String] objects, not [PSCustomObject] objects.
Only -? will behave identically to Get-help and will provide [PSCustomObject] objects.
To see what's going on, check the different output from :
help firewall | %{ $_.GetType() }
And
Get-help firewall | %{ $_.GetType() }
And, for cmdlet,
Select-Object -? | %{ $_.gettype() }

powershell string urldecoding execution time

Performing URLdecoding in CGI hybrid script by powershell oneliner:
echo %C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA| powershell.exe "Add-Type -AssemblyName System.Web;[System.Web.HttpUtility]::UrlDecode($Input) | Write-Host"
execution time of this oneliner is between 2-3 seconds on virtual machine. Is it because of .NET object is employed? Is there any way to decrease execution time? Have also some C written, lightning fast urldecode.exe utility, but unfortunatelly it does not eat STDIN.
A note if you're passing the input data as a string literal, as in the sample call in the question (rather than as output from a command):
If you're calling from an interactive cmd.exe session, %C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA works as-is - unless any of the tokens between paired % instances happen to be the name of an existing environment variable (which seems unlikely).
From a batch file, you'll have to escape the % chars. by doubling them - see this answer; which you can obtain by applying the following PowerShell operation on the original string:
'%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA' -replace '%', '%%'
Is it because of .NET object is employed?
Yes, powershell.exe, as a .NET-based application requires starting the latter's runtime (CLR), which is nontrivial in terms of performance.
Additionally, powershell.exe by default loads the initialization files listed in its $PROFILE variable, which can take additional time.
Pass the -NoProfile CLI option to suppress that.
Have also some C written, lightning fast urldecode.exe utility, but unfortunately it does not eat STDIN.
If so, pass the data as an argument, if feasible; e.g.:
urldecode.exe "%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA"
If the data comes from another command's output, you can use for /f to capture in a variable first, and then pass the latter.
If you do need to call powershell.exe, PowerShell's CLI, after all:
There's not much you can do in terms of optimizing performance:
Add -NoProfile, as suggested.
Pass the input data as an argument.
Avoid unnecessary calls such as Write-Host and rely on PowerShell's implicit output behavior instead.[1]
powershell.exe -NoProfile -c "Add-Type -AssemblyName System.Web;[System.Web.HttpUtility]::UrlDecode('%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA')"
[1] Optional reading: How do I output machine-parseable data from a PowerShell CLI call?
Note: The sample commands are assumed to be run from cmd.exe / outside PowerShell.
PowerShell's CLI only supports text as output, not also raw byte data.
In order to output data for later programmatic processing, you may have to explicitly ensure that what is output is machine-parseable rather than something that is meant for display only.
There are two basic choices:
Rely on PowerShell's default output formatting for outputting what are strings (text) to begin with, as well as for numbers - though for fractional and very large or small non-integer numbers additional effort may be required.
Explicitly use a structured, text-based data format, such as CSV or Json, to represent complex objects.
Rely on PowerShell's default output formatting:
If the output data is itself text (strings), no extra effort is needed. This applies to your case, and therefore simply implicitly outputting the string returned from the [System.Web.HttpUtility]::UrlDecode() call is sufficient:
# A simple example that outputs a 512-character string; note that
# printing to the _console_ (terminal) will insert line breaks for
# *display*, but the string data itself does _not_ contain any
# (other than a single _trailing one), irrespective of the
# console window width:
powershell -noprofile -c "'x' * 512"
If the output data comprises numbers, you may have to apply explicit, culture-invariant formatting if your code must run with different cultures in effect:
True integer types do not require special handling, as their string representation is in effect culture-neutral.
However, fractional numbers ([double], [decimal]) do use the current culture's decimal mark, which later processing may not expect:
# With, say, culture fr-FR (French) in effect, this outputs
# "1,2", i.e. uses a *comma* as the decimal mark.
powershell -noprofile -c "1.2"
# Simple workaround: Let PowerShell *stringify* the number explicitly
# in an *expandable* (interpolating) string, which uses
# the *invariant culture* for formatting, where the decimal
# mark is *always "." (dot).
# The following outputs "1.2", irrespective of what culture is in effect.
powershell -noprofile -c " $num=1.2; \"$num\" "
Finally, very large and very small [double] values can result in exponential notation being output (e.g., 5.1E-07 for 0.00000051); to avoid that, explicit number formatting is required, which can be done via the .ToString() method:
# The following outputs 0.000051" in all cultures, as intended.
powershell -noprofile -c "$n=0.000051; $n.ToString('F6', [cultureinfo]::InvariantCulture)"
More work is needed if you want to output representations of complex objects in machine-parseable form, as discussed in the next section.
Relying on PowerShell's default output formatting is not an option in this case, because implicit output and (equivalent explicit Write-Output calls) cause the CLI to apply for-display-only formatting, which is meaningful to the human observer but cannot be robustly parsed.
# Produces output helpful to the *human observer*, but isn't
# designed for *parsing*.
# `Get-ChildItem` outputs [System.IO.FileSystemInfo] objects.
powershell -noprofile -c "Get-ChildItem /"
Note that use of Write-Host is not an alternative: Write-Host fundamentally isn't designed for data output, and the textual representation it creates for complex objects are typically not even meaningful to the human observer - see this answer for more information.
Use a structured, text-based data format, such as CSV or Json:
Note:
Hypothetically, the simplest approach is to use the CLI's -OutputFormat Xml option, which serializes the output using the XML-based CLIXML format PowerShell itself uses for remoting and background jobs - see this answer.
However, this format is only natively understood by PowerShell itself, and for third-party applications to parse it they'd have to be .NET-based and use the PowerShell SDK.
Also, this format is automatically used for both serialization and deserialization if you call another PowerShell instance from PowerShell, with the command specified as a script block ({ ... }) - see this answer. However, there is rarely a need to call the PowerShell CLI from PowerShell itself, and direct invocation of PowerShell code and scripts provides full type fidelity as well as better performance.
Finally, note that all serialization formats, including CSV and JSON discussed below, have limits with respect to faithfully representing all aspects of the data, though -OutputFormat Xml comes closest.
PowerShell comes with cmdlets such as ConvertTo-Csv and ConvertTo-Json, which make it easy to convert output to the structured CSV and JSON formats.
Using a Get-Item call to get information about PowerShell's installation directory ($PSHOME) as an example; Get-Item outputs a System.IO.DirectoryInfo instance in this case:
Use of ConvertTo-Csv:
C:\>powershell -noprofile -c "Get-Item $PSHOME | ConvertTo-Csv -NoTypeInformation"
"PSPath","PSParentPath","PSChildName","PSDrive","PSProvider","PSIsContainer","Mode","BaseName","Target","LinkType","Name","FullName","Parent","Exists","Root","Extension","CreationTime","CreationTimeUtc","LastAccessTime","LastAccessTimeUtc","LastWriteTime","LastWriteTimeUtc","Attributes"
"Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\WindowsPowerShell\v1.0","Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\WindowsPowerShell","v1.0","C","Microsoft.PowerShell.Core\FileSystem","True","d-----","v1.0","System.Collections.Generic.List`1[System.String]",,"v1.0","C:\Windows\System32\WindowsPowerShell\v1.0","WindowsPowerShell","True","C:\",".0","12/7/2019 4:14:52 AM","12/7/2019 9:14:52 AM","3/14/2021 10:33:10 AM","3/14/2021 2:33:10 PM","11/6/2020 3:52:41 AM","11/6/2020 8:52:41 AM","Directory"
Note: -NoTypeInformation is no longer needed in PowerShell (Core) 7+
Using ConvertTo-Json:
C:\>powershell -noprofile -c "Get-Item $PSHOME | ConvertTo-Json -Depth 1"
{
"Name": "v1.0",
"FullName": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0",
"Parent": {
"Name": "WindowsPowerShell",
"FullName": "C:\\Windows\\System32\\WindowsPowerShell",
"Parent": "System32",
"Exists": true,
"Root": "C:\\",
"Extension": "",
"CreationTime": "\/Date(1575710092565)\/",
"CreationTimeUtc": "\/Date(1575710092565)\/",
"LastAccessTime": "\/Date(1615733476841)\/",
"LastAccessTimeUtc": "\/Date(1615733476841)\/",
"LastWriteTime": "\/Date(1575710092565)\/",
"LastWriteTimeUtc": "\/Date(1575710092565)\/",
"Attributes": 16
},
"Exists": true
// ...
}
Since JSON is a hierarchical data format, the serialization depth must be limited with -Depth in order to prevent "runaway" serialization when serializing arbitrary .NET types; this isn't necessary for [pscustomobject] and [hashtable] object graphs composed of primitive .NET types only.

"$xyz" and "Write-Host "$xyz"" giving different output

I am hashing all the files in one location, an origin folder, and writing the hashes to a variable and then doing the same to all the files in another location, a destination folder:
$origin = Get-ChildItem .\Test1 | Get-FileHash | Format-Table -Property Hash -HideTableHeaders
$destination = Get-ChildItem .\Test2 | Get-FileHash | Format-Table -Property Hash -HideTableHeaders
Then I am comparing them with Compare-Object like so:
Compare-Object $origin $destination
Now in my test I purposefully have deviations, so when the above code returned no differences I knew I had a problem.
Then I found out that if I do the following, that the hash values arn't there:
PS> Write-Host "$origin"
Microsoft.PowerShell.Commands.Internal.Format.FormatStartData Microsoft.PowerShell.Commands.Internal.Format.GroupStartData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData Microsoft.PowerShell.Commands.Internal.Format.GroupEndData Microsoft.PowerShell.Commands.Internal.Format.FormatEndData
However, if I just type the following and press enter, then the hash values are present (like I want):
PS> $origin
6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EAA22F1D49C01E52DDB7875B4B
D4735E3A265E16EEE03F59718B9B5D03019C07D8B6C51F90DA3A666EEC13AB35
4E07408562BEDB8B60CE05C1DECFE3AD16B72230967DE01F640B7E4729B49FCE
I am assuming when I use Compare-Object, that my variables are not presenting the hash values like I expected.
Does anyone know what is going on or have any recommendations? This is being used to ensure files are moved from an origin location to a destination location (this is one check in a script I'm working on). I am keeping this purely PowerShell, which means no xcopy or robocopy.
Re use of Format-Table to create the input collections for Compare-Object:
Only ever use Format-* cmdlets for display formatting; never use them if data must be programmatically processed.
Format-* cmdlets output formatting instructions, not data - see this answer.
Therefore:
Omit the Format-Table calls from your input-collection definition commands:
$origin=Get-ChildItem .\Test1 | Get-FileHash
$destination=Get-ChildItem .\Test2 | Get-FileHash
Then pass the names of the properties to compare the objects by to Compare-Object:
Compare-Object $origin $destination -Property Path, Hash
Note the need to compare by both path and hash, to make sure that only files of the same name are compared.
As an aside: If you didn't specify -Property, the objects would by default be compared by their .ToString() value - and since the Microsoft.PowerShell.Commands.FileHashInfo instances output by Get-FileHash only ever stringify to that very type name (irrespective of their specific property values), no differences would be found.
As for $origin vs. Write-Host $orgin:
Just executing $origin is implicitly like executing Write-Output $origin - it writes to the success output stream (see about_Redirection), which by default goes to the console.
Success output that goes to the console is automatically formatted by PowerShell's rich output-formatting system.
Write-Host, by contrast, serves a different purpose than Write-Output:
It writes directly to the console[1], bypassing PowerShell's success output stream and thereby also its usual formatting. Its primary purpose is to write status messages, interactive prompt messages, ... to the display - as opposed to outputting data.
Write-Host itself applies output formatting, but only by simple .ToString() stringification, which often yields unhelpful (type name-only) representations, as in your case.
See this answer for more information about the differences between Write-Output and Write-Host.
[1] Technically, since PowerShell version 5, Write-Host output reaches the console via the information output stream (number 6), but its primary purpose is still to write to the display as opposed to outputting data.

How can I move from Windows traditional command line to the modern PowerShell?

I was used to a few command line tricks in Windows that increased my productivity a lot.
Now I am told that I should move to PowerShell because it's more POWERful. It took me a while to get a little bit hang of it (objects, piping, etc.), and there are a lot of great tutorials on how to get a few things done.
However, some (relatively) basic trick still puzzle me. For instance, what is the equivalent of the FOR structure in PowerShell?
For example,
FOR %i IN (*.jpg) DO Convert %i -resize 800x300 resized/%i
The above line takes all of photos in a folder and uses the ImageMagick's Convert tool to resize the images and restores the resized imaged in a sub-folder called RESIZED.
In PowerShell I tried the command:
Dir ./ | foreach {convert $_.name -resize 800x300 resized/$_name}
This can't work despite all of the googling around I did. What is missing?
Note that / rather than \ is used as the path separator in this answer, which works on Windows too and makes the code compatible with the cross-platform PowerShell Core editions.
tl;dr:
$convertExe = './convert' # adjust path as necessary
Get-ChildItem -File -Filter *.jpg | ForEach-Object {
& $convertExe $_.Name -resize 800x300 resized/$($_.Name)
}
Read on for an explanation and background information.
The equivalent of:
FOR %i IN (*.jpg)
is:
Get-ChildItem -File -Filter *.jpg
or, with PowerShell's own wildcard expressions (slower, but more powerful):
Get-ChildItem -File -Path *.jpg # specifying parameter name -Path is optional
If you're not worried about restricting matches to files (as opposed to directories), Get-Item *.jpg will do too.
While dir works as a built-in alias for Get-ChildItem, I recommend getting used to PowerShell's own aliases, which follow a consistent naming convention; e.g., PowerShell's own alias for Get-ChildItem is gci
Also, in scripts it is better to always use the full command names - both for readability and robustness.
As you've discovered, to process the matching files in a loop you must pipe (|) the Get-ChildItem command's output to the ForEach-Object cmdlet, to which you pass a script block ({ ... }) that is executed for each input object, and in which $_ refers to the input object at hand.
(foreach is a built-in alias for ForEach-Object, but note that there's also a foreach statement, which works differently, and it's important not to confuse the two.)
There are 2 pitfalls for someone coming from the world of cmd.exe (batch files):
In PowerShell, referring to an executable by filename only (e.g., convert) does not execute an executable by that name located in the current directory, for security reasons.
Only executables in the PATH can be executed by filename only, and unless you've specifically placed ImageMagick's convert.exe in a directory that comes before the SYSTEM32 directory in the PATH, the standard Windows convert.exe utility (whose purpose is to convert FAT disk volumes to NTFS) will be invoked.
Use Get-Command convert to see what will actually execute when you submit convert; $env:PATH shows the current value of the PATH environment variable (equivalent of echo %PATH%).
If your custom convert.exe is indeed in the current directory, invoke it as ./convert - i.e., you must explicitly reference its location.
Otherwise (your convert.exe is either not in the PATH at all or is shadowed by a different utility) specify the path to the executable as needed, but note that if you reference that path in a variable or use a string that is single- or double-quoted (which is necessary if the path contains spaces, for instance), you must invoke with &, the call operator; e.g.,
& $convertExe ... or & "$HOME/ImageMagic 2/convert" ...
PowerShell sends objects through the pipeline, not strings (this innovation is at the heart of PowerShell's power). When you reference and object's property or an element by index as part of a larger string, you must enclose the expression in $(...), the subexpression operator:
resized/$($_.Name) - Correct: property reference enclosed in $(...)
resized/$_.Name - !! INCORRECT - $_ is stringified on its own, followed by literal .Name
However, note that a stand-alone property/index reference or even method call does not need $(...); e.g., $_.Name by itself, as used in the command in the question, does work, and retains its original type (is not stringified).
Note that a variable without property / index access - such as $_ by itself - does not need $(...), but in the case at hand $_ would expand to the full path. For the most part, unquoted tokens that include variable references are treated like implicitly double-quoted strings, whose interpolation rules are summarized in this answer of mine; however, many additional factors come into play, which are summarized here; edge cases are highlighted in this question.
At the end of the day, the safest choice is to double-quote strings that contain variable references or subexpressions:
"resized/$($_.Name)" - SAFEST
Use:
Get-ChildItem | foreach {convert $_.name -resize 800x300 resized/$($_.name)}
Or, perhaps, you need to pass the full name (with path), also showing a shorter syntax (using aliases):
gci | % {convert $_.fullname -resize 800x300 resized/$($_.name)}
Also, you might want to supply the full path to the executable.
Revised based on comments given below
There are many applications with the name "Convert". If I do
Get-Command Convert
on my computer. It shows me an app that is part of the Windows system. If PowerShell is running the wrong app on you, it's never going to work.
The solution will be to point PowerShell at the convert tool inside the ImageMagick program folder. A Google search on "ImageMagick PowerShell" will lead you to lots of people who have faced the same problem as you.

PowerShell: programmatically access script documentation

Is there a way to programmatically load the documentation of a .ps1 script file outside of commands like get-help? In other words, can text defined under .SYNOPSIS, .DESCRIPTION, etc. be accessed programmatically other than filtering the string output of get-help itself?
Among other things, I'm trying to find where I have gaps in documentation coverage in my script library. I'd also like to be able to display lists of certain scripts with their synopsis attached.
Yes, those are all accessible. Get-Help returns (just like any other cmdlet) an object, and the default rendering of that object is what you see in the console.
However, if you pump get-help's output through format-list, like this:
get-help get-childitem | format-list
You'll get a list of name-value pairs of the properties. To get the synopsis, you can do the following:
get-help get-childitem |select-object -property synopsis
And the output:
Synopsis
--------
Gets the files and folders in a file system drive.
If your .ps1 file has no cmdlets defined in it (your comment-based help covers the whole script), get-help file.ps1|select synopsis should work. Otherwise, you'll need to "dot-source" the files to load the cmdlet definitions into memory, then use get-help as above.