I am hashing all the files in one location, an origin folder, and writing the hashes to a variable and then doing the same to all the files in another location, a destination folder:
$origin = Get-ChildItem .\Test1 | Get-FileHash | Format-Table -Property Hash -HideTableHeaders
$destination = Get-ChildItem .\Test2 | Get-FileHash | Format-Table -Property Hash -HideTableHeaders
Then I am comparing them with Compare-Object like so:
Compare-Object $origin $destination
Now in my test I purposefully have deviations, so when the above code returned no differences I knew I had a problem.
Then I found out that if I do the following, that the hash values arn't there:
PS> Write-Host "$origin"
Microsoft.PowerShell.Commands.Internal.Format.FormatStartData Microsoft.PowerShell.Commands.Internal.Format.GroupStartData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData Microsoft.PowerShell.Commands.Internal.Format.GroupEndData Microsoft.PowerShell.Commands.Internal.Format.FormatEndData
However, if I just type the following and press enter, then the hash values are present (like I want):
PS> $origin
6B86B273FF34FCE19D6B804EFF5A3F5747ADA4EAA22F1D49C01E52DDB7875B4B
D4735E3A265E16EEE03F59718B9B5D03019C07D8B6C51F90DA3A666EEC13AB35
4E07408562BEDB8B60CE05C1DECFE3AD16B72230967DE01F640B7E4729B49FCE
I am assuming when I use Compare-Object, that my variables are not presenting the hash values like I expected.
Does anyone know what is going on or have any recommendations? This is being used to ensure files are moved from an origin location to a destination location (this is one check in a script I'm working on). I am keeping this purely PowerShell, which means no xcopy or robocopy.
Re use of Format-Table to create the input collections for Compare-Object:
Only ever use Format-* cmdlets for display formatting; never use them if data must be programmatically processed.
Format-* cmdlets output formatting instructions, not data - see this answer.
Therefore:
Omit the Format-Table calls from your input-collection definition commands:
$origin=Get-ChildItem .\Test1 | Get-FileHash
$destination=Get-ChildItem .\Test2 | Get-FileHash
Then pass the names of the properties to compare the objects by to Compare-Object:
Compare-Object $origin $destination -Property Path, Hash
Note the need to compare by both path and hash, to make sure that only files of the same name are compared.
As an aside: If you didn't specify -Property, the objects would by default be compared by their .ToString() value - and since the Microsoft.PowerShell.Commands.FileHashInfo instances output by Get-FileHash only ever stringify to that very type name (irrespective of their specific property values), no differences would be found.
As for $origin vs. Write-Host $orgin:
Just executing $origin is implicitly like executing Write-Output $origin - it writes to the success output stream (see about_Redirection), which by default goes to the console.
Success output that goes to the console is automatically formatted by PowerShell's rich output-formatting system.
Write-Host, by contrast, serves a different purpose than Write-Output:
It writes directly to the console[1], bypassing PowerShell's success output stream and thereby also its usual formatting. Its primary purpose is to write status messages, interactive prompt messages, ... to the display - as opposed to outputting data.
Write-Host itself applies output formatting, but only by simple .ToString() stringification, which often yields unhelpful (type name-only) representations, as in your case.
See this answer for more information about the differences between Write-Output and Write-Host.
[1] Technically, since PowerShell version 5, Write-Host output reaches the console via the information output stream (number 6), but its primary purpose is still to write to the display as opposed to outputting data.
Related
I have this working, but need LastWriteTime and can't get it.
Get-ChildItem -Recurse | Select-String -Pattern "CYCLE" | Select-Object Path, Line, LastWriteTime
I get an empty column and zero Date-Time data
Select-String's output objects, which are of type Microsoft.PowerShell.Commands.MatchInfo, only contain the input file path (string), no other metadata such as LastWriteTime.
To obtain it, use a calculated property, combined with the common -PipelineVariable parameter,
which allows you to reference the input file at hand in the calculated property's expression script block as a System.IO.FileInfo instance as output by Get-ChildItem, whose .LastWriteTime property value you can return:
Get-ChildItem -File -Recurse -PipelineVariable file |
Select-String -Pattern "CYCLE" |
Select-Object Path,
Line,
#{
Name='LastWriteTime';
Expression={ $file.LastWriteTime }
}
Note how the pipeline variable, $file, must be passed without the leading $ (i.e. as file) as the -PipelineVariable argument . -PipelineVariable can be abbreviated to -pv.
LastWriteTime is a property of System.IO.FileSystemInfo, which is the base type of the items Get-ChildItem returns for the Filesystem provider (which is System.IO.FileInfo for files). Path and Line are properties of Microsoft.PowerShell.Commands.MatchInfo, which contains information about the match, not the file you passed in. Select-Object operates on the information piped into it, which comes from the previous expression in the pipeline, your Select-String in this case.
You can't do this as a (well-written) one-liner if you want the file name, line match, and the last write time of the actual file to be returned. I recommend using an intermediary PSCustomObject for this and we can loop over the found files and matches individually:
# Use -File to only get file objects
$foundMatchesInFiles = Get-ChildItem -Recurse -File | ForEach-Object {
# Assign $PSItem/$_ to $file since we will need it in the second loop
$file = $_
# Run Select-String on each found file
$file | Select-String -Pattern CYCLE | ForEach-Object {
[PSCustomObject]#{
Path = $_.Path
Line = $_.Line
FileLastWriteTime = $file.LastWriteTime
}
}
}
Note: I used a slightly altered name of FileLastWriteTime to exemplify that this comes from the returned file and not the match provided by Select-String, but you could use LastWriteTime if you wish to retain the original property name.
Now $foundMatchesInFiles will be a collection of files which have CYCLE occurring within them, the path of the file itself (as returned by Select-String), and the last write time of the file itself as was returned by the initial Get-ChildItem.
Additional considerations
You could also use Select-Object and computed properties but IMO the above is a more concise approach when merging properties from unrelated objects together. While not a poor approach, Select-Object outputs data with a type containing the original object type name (e.g. Selected.Microsoft.PowerShell.Commands.MatchInfo). The code may work fine but can cause some confusion when others who may consume this object in the future inspect the output members. LastWriteTime, for example, belongs to FileSystemInfo, not MatchInfo. Another developer may not understand where the property came from at first if it has the MatchInfo type referenced. It is generally a better design to create a new object with the merged properties.
That said this is a minor issue which largely comes down to stylistic preference and whether this object might be consumed by others aside from you. I write modules and scripts that many other teams in my organization consume so this is a concern for me. It may not be for you. #mklement0's answer is an excellent example of how to use computed properties with Select-Object to achieve the same functional result as this answer.
There are a lot of questions and answers about comparing the hash of two folder for integrity like this one. Assuming I have a folder that I copied to a backup medium (External drive, flash, optical disc) and would like to delete the original to save space.
What is the best way to save the original's folder hashes (before deletion) in a text file perhaps and check the backup's integrity much later against that file.
Note that if you delete the originals first and later find that the backup lacks integrity, so to speak, all you'll know is that something went wrong; the non-corrupted data will be gone.
You can create a CSV file with 2 columns, RelativePath (the full file path relative to the input directory) and Hash, and save it to a CSV file with Export-Csv:
$inputDir = 'C:\path\to\dir' # Note: specify a *full* path.
$prefixLen = $inputDir.Length + 1
Get-ChildItem -File -Recurse -LiteralPath $inputDir |
Get-FileHash |
Select-Object #{
Name='RelativePath'
Expression={ $_.Path.Substring($prefixLen) }
},
Hash |
Export-Csv originalHashes.csv -NoTypeInformation -Encoding utf8
Note: In PowerShell (Core) 7+, neither -NoTypeInformation nor -Encoding utf8 are needed, though note that the file will have no UTF-8 BOM; use -Encoding utf8bom if you want one; conversely, in Windows PowerShell you invariably get a BOM.
Note:
The Microsoft.PowerShell.Commands.FileHashInfo instances output by Get-FileHash also have an .Algorithm property naming the hashing algorithm that was used ('SHA256' by default, or as specified via the -Algorithm parameter).
If you want this property included (whose value will be the same for all CSV rows), you simply add Algorithm to the array of properties passed to Select-Object above.
Note how a hashtable (#{ ... }) passed as the second property argument to Select-Object serves as a calculated property that derives the relative path from each .Path property value (which contains the full path).
You can later apply the same command to the backup directory tree, saving to, say, backupHashes.csv, and compare the two CSV files with Compare-Object:
Compare-Object (Import-Csv -LiteralPath originalHashes.csv) `
(Import-Csv -LiteralPath backupHashes.csv) `
-Property RelativePath, Hash
Note: There's no strict need to involve files in the operation - one or both output collections can be captured in memory and can be used directly in the comparison - just omit the Export-Csv call in the command above and save to a variable ($originalHashes = Get-ChildItem ...)
This basically works
foreach ($cprev in $CopyPreventeds) {
Write-Host ("prevented copy $(($cprev)."Name")")
$cprev | Select-Object Path, Name, Length, LastWrite, DestinationNewer | Export-Csv '.\prevented.csv' -NoTypeInformation
}
But only the last output is written to the csv. How could I write all contents to a new csv with an output at the same time for the user in PowerShell.
Maybe I'm missing something?
While I appreciate a solution has already been proposed in the comments, I have to ask, given the narrow scope of the question why are we using an obscure, albeit clever technique? And/or, repeatedly invoking Export-Csv...
The question doesn't mention sparing a variable. Moreover, There doesn't appear to be a need for the ForEach loop.
$CopyPreventeds |
Select-Object Path, Name, Length, LastWrite, DestinationNewer |
Export-Csv '.\prevented.csv' -NoTypeInformation
In the above $CopyPreventeds already exists and remains so, unmolested after the export. You would need only to output it again for the benefit of an interactive user. All taking advantage of PowerShell's intuitive pipeline and features.
Moreover, since the iteration variable $cprev isn't needed you are still less one variable.
Note: You don't need -Append because you are streaming into a single Export-Csv command, as opposed to repeatedly invoking it.
There are at least 2 ways (probably many more) you could conveniently output to an interactive user.
1: Echo a header, something like "The following copies were prevented:" then echo the variable $CopyPreventeds, presumably to a table.
Note: That given multiple points at which you seem only interested in a subset of properties. You may think about trimming those objects beforehand:
$CopyPreventeds =
$CopyPreventeds |
Select-Object Path, Name, Length, LastWrite, DestinationNewer
$CopyPreventeds | Export-Csv '.\prevented.csv' -NoTypeInformation
Write-Host "The following copies were prevented:"
$CopyPreventeds | Format-Table -AutoSize | Out-Host
Note: More than 4 Properties in a [PSCustomObject] (resulting from Select-Object) where custom formatting hasn't been defined will by default output as a list, so use Format-Table to overcome that. Out-Host is then used to prevent pipeline pollution.
2: Return to using a ForEach-Object Loop for the output between the Select-Object and the Export-Csv command.
$CopyPreventeds |
Select-Object Path, Name, Length, LastWrite, DestinationNewer
ForEach-Object{
"Prevented Copy : {0}, {1}, {2}, {3}, {4}" -f $_.Path, $_.Name, $_.Length, $_.LastWrite, $_.DestinationNewer |
Write-Host
$_
} |
Export-Csv '.\prevented.csv' -NoTypeInformation
In this example, when you are done outputting to the screen (admittedly a little messy), you emit $_ from the loop, thus piping it to Export-Csv just the same.
Note: there are a number of ways to construct strings, I choose to use the -f operator here because it's a little cleaning than imbedding numerous $() sub expressions. And, of course this assume you want to prefix on every line Which I personally think is gratuitous, so I'd choose something more like #1..
Currently my log function spits out the information in a single column and is hard to read. Is there a way to make it split up into different columns which each (DisplayName, PoolName, PoolSnapshot, and DesktopSVIVmSnapshot) and its respective information is put correctly?
function log ([string]$entry) {
Write-Output $entry | Out-File -Append "C:\logs\SNAPSHOT.csv"
}
Add-PSSnapin Quest.ActiveRoles.ADManagement
$date = Get-Date -Format "MM-dd-yyyy"
$time = Get-Date -Format "hh:mm:sstt"
# begin log
log $(Get-Date)
log "The below Desktops are not using the correct Snapshot."
if (#($DesktopExceptions).Count -lt 1) {
Write-Output "All desktops in $pool are currently using the correct snapshots." |
Out-File -Append "C:\logs\SNAPSHOT.csv"
} else {
Write-Output $DesktopExceptions |
Select-Object DisplayName,PoolName,PoolSnapshot,DesktopSVIVmSnapshot |
sort DisplayName |
Out-File -Append "C:\logs\SNAPSHOT.csv"
}
log $(Get-Date)
09/11/2017 12:16:17
DisplayName PoolName PoolSnapshot DesktopSVIVmSnapshot
----------- -------- ------------ --------------------
xxxc-13v xxxc-xxx /8-11-2017/09-07-2017 /8-11-2017
xxxc-15v xxxc-xxx /8-11-2017/09-07-2017 /8-11-2017
xxxc-1v xxxc-xxx /8-11-2017/09-07-2017 /8-11-2017
xxxc-20v xxxc-xxx /8-11-2017/09-07-2017 /8-11-2017
Note: I removed parts of the log for in the hopes to not make the post long.
CSV files require uniform lines: a header line with column names, followed by data lines containing column values.
By writing the output from Get-Date first - a single date/time string - followed by another single-string output, followed by multi-column output from your $DesktopExceptions | Select-Object ... call, you're by definition not creating a valid CSV file.
If you still want to create such a file:
log (Get-Date) # With a single command, you don't need $(...) - (...) will do.
log "The below Desktops are not using the correct Snapshot."
If ($DesktopExceptions) # a non-empty array / non-$null object
{
log ($DesktopExceptions |
Select-Object DisplayName,PoolName,PoolSnapshot,DesktopSVIVmSnapshot |
Sort-Object DisplayName |
ConvertTo-Csv -NoTypeInformation)
}
Else
{
log "All desktops in $pool are currently using the correct snapshots."
}
log (Get-Date)
By defining your log() function's parameter as type [string], you're effectively forcing stringification of whatever object you pass to it. This stringification is the same you get when you embed a variable reference or command inside "..." (string expansion / interpolation) - but it is not the same as what you get by default, when you print to the console.
Out-File, by contrast, does result in the same output you get when printing to the console, which, however, is a format for human consumption, not for machine parsing (as CSV is, for instance).
To get CSV-formatted output, you must either use Export-Csv - to write directly to a file - or ConvertTo-Csv- to get a string representation.
Also note that there's typically no reason to use Write-Output explicitly - any command / expression's output that is not explicitly assigned to a variable / redirected (to a file or $null) is implicitly sent to PowerShell's [success] output stream; e.g., Write-Output Get-Date is the same as just Get-Date.
It looks like you're just writing an object, and taking the default PowerShell formatter behavior.
A better thing to do is make your log only responsible for one thing - writing messages to a file (no formatting). Here's an example of what you might try:
function Write-LogMessage {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, HelpMessage = "The text-content to write to the log file.",
ValueFromPipeline = $true)]
[string]$Text
)
Process {
Write-Host -ForegroundColor Green $Text
}
}
Set-Alias log Write-LogMessage
Note: This example writes directly to the PowerShell console, but you would in practice need to direct output to a file (using Out-File or one of the redirection operators - see Get-Help about_Operators).
To use it, you would write something like this:
"This is a message that would be written" | Write-LogMessage
For your specific example, you could just format the message inline, and pipe it:
Write-Output $DesktopExceptions | Select-Object DisplayName,PoolName,PoolSnapshot,DesktopSVIVmSnapshot | sort DisplayName | ForEach-Object { "{0}: Host = {1}, Pool = {2}, Pool SN = {3}, SVIV Snapshot = {4}" -f (Get-Date), $_.DisplayName, $_.PoolName, $_.PoolSnapshot, $_.DesktopSVIVmSnapshot } | log
Note that you don't need the log statement: just add formatting before piping to the Out-File cmdlet, and you'll get what you're after.
Edit: The OP asked in the original post how to format columns (tabular output). To achieve this, you can use either the ConvertTo-Csv or Export-Csv cmdlets (generally, you would use the -NoTypeInformation switch parameter with these commands, to avoid the first line of the output being a type definition). An example of this is:
$DesktopExceptions | Select-Object DisplayName,PoolName,PoolSnapshot,DesktopSVIVmSnapshot | sort DisplayName | Export-Csv C:\Temp\Datum.csv -NoTypeInformation
As pointed out in another answer, using Write-Output is not required, because PowerShell automatically writes all output to the output stream unless otherwise directed (using file redirection, a redirection operator, or the Out-Null cmdlet).
Please read my answer as part solution and part advice.
The "problem" with PowerShell is that it doesn't capture only the output of your code. It will capture output from other scripts, modules and executables. In other words, any attempt to make logging behave like it's generated by e.g. C# with NLOG, has an inherent problem.
I looked into this subject myself for a complex continuous delivery pipeline I'm building. I understood that a structured log will not be 100% possible and therefore I accepted the purpose of PowerShell transcription (Start-Transcript). But still I wanted to avoid creating functions like Write-Log and if possible provide an enhanced output for all code that uses Write-Debug, Write-Verbose functionality.
I ended up creating XWrite PowerShell module which works very well, even to my own suprize. I use it because it enhances the produced trace message by the caller's name (cmdlet or script) and a timestamp. The caller's name helps a lot with troubleshooting and the timestamp I use to implicitly benchmark. here are a couple of example
DEBUG: Test-MyXWrite.ps1: Hello
DEBUG: Script: Test-MyXWrite.ps1: 20170804: 10:57:27.845: Hello
There are some limitations though. Any binary's code trace output will not be enhanced. Also if a cmllet refers explicitly to the Write-* using their full namespace it will not work. To capture line by line all trace and output requires some very deep into the .net types of PowerShell implementation hooking. There is a guy who has done this, but I don't want to get influence the PowerShell process's behavior that aggresively. And at this moment I believe that to be the role of the transcription.
If you like the idea, install the module from XWrite
At some point, I would like to extend the module with a redirection to telemetry services, but I've still not decided I want to do that, because I will not capture the above mentioned exceptions and other executable. It will just offer me visible progress as the script is executing.
I am sorting a large directory of files and I am trying to select individual lines from the output of an ls command and show those only, but I get weird results and I am not familiar enough with powershell to know what I'm doing wrong.
this approach works:
ls > data.txt
select-string 2012 data.txt
rm data.txt
but it seems wasteful to me to create a file just to read the data that I already have to fill into the file. I want to pipe the output directly to select-string.
I have tried this approach:
ls | select-string 2012
but that does not give me the appropriate output.
My guess is that I need to convert the output from ls into something select-string can work with, but I have no idea how to do that, or even whether that is actually the correct approach.
PowerShell is object-oriented, not pure text like cmd. If you want to get fileobjects(lines) that were modified in 2012, use:
Get-ChildItem | Where-Object { $_.LastWriteTime.Year -eq 2012 }
If you want to get fileobjects with "2012" in the filename, try:
Get-ChildItem *2012*
When you use
ls | select-string 2012
you're actually searching for lines with "2012" INSIDE every file that ls / get-childitem listed.
If you really need to use select-string on the output from get-childitem, try converting it to strings, then splitting up into lines and then search it. Like this:
(Get-ChildItem | Out-String) -split "`n" | Select-String 2012
I found another simple way to convert objects to strings:
Get-ChildItem | Out-String -stream | Select-String 2012
in this very interesting article:
http://blogs.msdn.com/b/powershell/archive/2006/04/25/how-does-select-string-work-with-pipelines-of-objects.aspx
If you wanted Select-String to work on the Monad formatted output, you'll need to get that as a string. Here is the thing to grok about
our outputing. When your command sequence emits a stream of strings,
we emit it without processing. If instead, your command sequence
emits a stream of objects, then we redirect those objects to the
command Out-Default. Out-Default looks at the type of the object and
the registered formating metadata to see if there is a default view
for that object type. A view defines a FORMATTER and the metadata for
that command. Most objects get vectored to either Format-Table or
Format-List (though they could go to Format-Wide or Format-Custom).
THESE FORMATTERS DO NOT EMIT STRINGS! You can see this for yourself
by the following: "These formating records are then vectored to an
OUT-xxx command to be rendered into the appropriate data for a
particular output device. By default, they go to Out-Host but you can
pipe this to Out-File, Out-Printer or Out-String. (NOTE: these
OUT-xxx commands are pretty clever, if you pipe formating objects to
them, they'll render them. If you pipe raw object to them, they'll
first call the appropriate formatter and then render them.)