There are a lot of questions and answers about comparing the hash of two folder for integrity like this one. Assuming I have a folder that I copied to a backup medium (External drive, flash, optical disc) and would like to delete the original to save space.
What is the best way to save the original's folder hashes (before deletion) in a text file perhaps and check the backup's integrity much later against that file.
Note that if you delete the originals first and later find that the backup lacks integrity, so to speak, all you'll know is that something went wrong; the non-corrupted data will be gone.
You can create a CSV file with 2 columns, RelativePath (the full file path relative to the input directory) and Hash, and save it to a CSV file with Export-Csv:
$inputDir = 'C:\path\to\dir' # Note: specify a *full* path.
$prefixLen = $inputDir.Length + 1
Get-ChildItem -File -Recurse -LiteralPath $inputDir |
Get-FileHash |
Select-Object #{
Name='RelativePath'
Expression={ $_.Path.Substring($prefixLen) }
},
Hash |
Export-Csv originalHashes.csv -NoTypeInformation -Encoding utf8
Note: In PowerShell (Core) 7+, neither -NoTypeInformation nor -Encoding utf8 are needed, though note that the file will have no UTF-8 BOM; use -Encoding utf8bom if you want one; conversely, in Windows PowerShell you invariably get a BOM.
Note:
The Microsoft.PowerShell.Commands.FileHashInfo instances output by Get-FileHash also have an .Algorithm property naming the hashing algorithm that was used ('SHA256' by default, or as specified via the -Algorithm parameter).
If you want this property included (whose value will be the same for all CSV rows), you simply add Algorithm to the array of properties passed to Select-Object above.
Note how a hashtable (#{ ... }) passed as the second property argument to Select-Object serves as a calculated property that derives the relative path from each .Path property value (which contains the full path).
You can later apply the same command to the backup directory tree, saving to, say, backupHashes.csv, and compare the two CSV files with Compare-Object:
Compare-Object (Import-Csv -LiteralPath originalHashes.csv) `
(Import-Csv -LiteralPath backupHashes.csv) `
-Property RelativePath, Hash
Note: There's no strict need to involve files in the operation - one or both output collections can be captured in memory and can be used directly in the comparison - just omit the Export-Csv call in the command above and save to a variable ($originalHashes = Get-ChildItem ...)
Related
Editor's note:
The gist of this question is:
* How do I add custom properties to objects output by Get-ChildItem that contain derived path information, namely the parent folder path (as Folder and name (as Foldername)?
* How do I export the resulting objects to a CSV file?
Currently have a script running that I took from StackOverflow and modified for my own use. The purpose of the script is to look at a directory, grab file names, then export them to a directory as a .csv file.
I was able to modify the script to pull through just the file name as originally it only pulled through the path, user etc. Was able to do this by adding ,Name. For some reason I can't get the script to pull through the parent folder it's in, also wanting the parents parent folder.
I believe this part of the code is what I am having most trouble with. Upon the select I was able to add ,Name but adding ,folder or ,foldername after the select doesn't pull through properly.
ForEach-Object {$_ | Add-Member -Name "Owner" -MemberType NoteProperty -Value (Get-Acl $_.FullName).Owner -PassThru} |
Sort-Object fullname |
Select FullName,CreationTime,LastWriteTime,Owner,Name,Folder,Foldername
The [System.IO.FileInfo] instances returned by Get-ChildItem for files do not have Folder or FolderName properties.
Get-ChildItem -File $HOME\Desktop | Get-Member, for instance, will show you the available properties, and will show you that the desired information can be derived from the PSPath and PSParentPath properties.
Select-Object allows hashtable-based property definitions, so-called calculated properties, which allow you to rename and/or transform properties and/or add custom properties derived from other property values by way of a script block.
Note: You can also use calculated properties with the Format-Table and Format-List cmdlets for creating output for display only (as well as with other cmdlets - see the linked help topic).
A simplified example of what you're looking for (includes output to a CSV file):
Get-ChildItem $HOME\Desktop | Select-Object Name,
#{ Name = 'Folder'; Expression = { Convert-Path $_.PSParentPath } },
#{ Name = 'Foldername'; Expression = { ($_.PSPath -split '\\')[-2] } } |
Export-Csv Out.csv -Encoding Utf8 -NoTypeInformation
Note that, alternatively, you could add Folder and FolderName properties to the input objects via Add-Member, as you did with the Owner property in your question.
Explanation:
Note that you can get more detailed information about any of the commands mentioned by running Get-Help <command-name> -Full; add -online to view the help topic in a browser; to learn more about the -split operator, run Get-Help about_split; to learn about PowerShell's help system in general, run Get-Help Get-Help -online.
Each #{ ... } construct passed to Select-Object is a hash table that defines a calculated property to attach to each output object:
The hash table must have two entries:
Name or Label, which defines the property's name; for brevity, you may use a (case-insensitive) prefix of the key name, such as just n or l.
Expression, which defines the property's value; again, a (case-insensitive) prefix of the key name works too, such as just e.
The expression can be a mere property name (a string), in case you simply want to rename an input property, but is more typically a script block ({ ... }), which is a piece of code that gets executed for each input object, and whose output becomes the value of the property being defined; inside that script block, automatic variable $_ (or $PSItem) refers to the input object at hand.
Definition of the Folder property: Convert-Path $_.PSParentPath converts the fully qualified PowerShell path that the PSParentPath property contains - which includes a prefix identifying the drive provider - to a regular filesystem path; e.g., Microsoft.PowerShell.Core\FileSystem::C:\Users\jdoe\Desktop -> C:\Users\jdoe\Desktop.
Definition of the Foldername property: ($_.PSPath -split '\\')[-2] splits the full path into components by path separator \, and then accesses the next-to-last component (-2), which is the parent folder name; e.g., C:\Users\jdoe\Desktop\file.txt -> Desktop
'\\' must be used to represent \, because -split's 1st RHS operand is a regular expression, where \ has special meaning and must therefore be doubled to be taken as a literal.
If you wanted to support / as the path separator as well for cross-platform support, you'd use ($_.PSPath -split '[\\/]')[-2].
Export-Csv exports the objects output by Select-Object to CSV Out.csv, where the input objects' property names become the header row, and the property values the data rows.
-Encoding Utf8, required in Windows PowerShell only (PowerShell (Core) 7+ now fortunately consistently uses BOM-less UTF-8), ensures that non-ASCII characters are properly encoded; by default, Export-Csv uses ASCII encoding and simply replaces non-ASCII characters, such as foreign letters, with verbatim ? characters, resulting in loss of information; note that -Encoding Utf8 in Windows PowerShell invariably creates UTF-8 files with a BOM.
-NoTypeInformation, again required in Windows PowerShell only, suppresses a line that Export-Csv by defaults adds as the first line of the output file, which contains the full type name (class name) of the input objects (e.g., #TYPE System.Management.Automation.PSCustomObject; this is meant to facilitate later reconversion into objects).
i have csv file with 3 columns SID, SamAccount name, ENABLED.
i also have folder containing files that called in a combination of "UVHD-"+SID.
i try to update the csv file with Length, LastWriteTime
so it will be like this for example:
SID SAMAccountName Enabled Length LastWriteTime
S-... FelixR False 205520896 02/02/2021 9:13:40
i tried many things and all failed
this is the best i could get:
Import-Csv $path\SID-ListTEST2.csv | select -ExpandProperty SID | ForEach-Object { Get-Childitem –Path $path\"UVHD-"$_.vhdx | Export-Csv $path\SID-ListTEST2.csv -Append | where $_ }
Use calculated properties:
(
Import-Csv $path\SID-ListTEST2.csv |
Select-Object *,
#{
Name='LastWriteTime';
Expression={ (Get-Item "$path\UVHD-$($_.SID).vhdx").LastWriteTime }
}
) | # Export-Csv -NoTypeInformation -Encoding utf8 $path\SID-ListTEST2.csv
Outputs to the display; remove the # from the last line to export to a CSV file instead.
Note the (...) around the pipeline, which ensures that all output is collected up front, which is the prerequisite for saving the results back to the original input file. Note that the original character encoding isn't necessarily preserved - use -Encoding to specify the desired one.
This adds one additional property, LastWriteTime; construct the other ones analogously.
For improved performance, you could cache the result of the Get-Item call, so that it doesn't have to be repeated in every calculated property: In the simplest case, use ($script:file = Get-Item ...) in the first calculated property, which you can then reuse as $script:file (or just $file) in the subsequent ones. Note that the $script: scope modifier is necessary, because the script blocks of calculated properties run in child scopes.[1]
Note that if no matching file exists, the Get-Item call fails silently and defaults to $null.
[1] Therefore, the more robust - but more cumbersome - approach would be to use Set-Variable -Scope 1 file (Get-Item ...) instead of $script:file = Get-Item ..., to ensure that the variable is created in the immediate parent scope, whatever it happens to be.
I have this working, but need LastWriteTime and can't get it.
Get-ChildItem -Recurse | Select-String -Pattern "CYCLE" | Select-Object Path, Line, LastWriteTime
I get an empty column and zero Date-Time data
Select-String's output objects, which are of type Microsoft.PowerShell.Commands.MatchInfo, only contain the input file path (string), no other metadata such as LastWriteTime.
To obtain it, use a calculated property, combined with the common -PipelineVariable parameter,
which allows you to reference the input file at hand in the calculated property's expression script block as a System.IO.FileInfo instance as output by Get-ChildItem, whose .LastWriteTime property value you can return:
Get-ChildItem -File -Recurse -PipelineVariable file |
Select-String -Pattern "CYCLE" |
Select-Object Path,
Line,
#{
Name='LastWriteTime';
Expression={ $file.LastWriteTime }
}
Note how the pipeline variable, $file, must be passed without the leading $ (i.e. as file) as the -PipelineVariable argument . -PipelineVariable can be abbreviated to -pv.
LastWriteTime is a property of System.IO.FileSystemInfo, which is the base type of the items Get-ChildItem returns for the Filesystem provider (which is System.IO.FileInfo for files). Path and Line are properties of Microsoft.PowerShell.Commands.MatchInfo, which contains information about the match, not the file you passed in. Select-Object operates on the information piped into it, which comes from the previous expression in the pipeline, your Select-String in this case.
You can't do this as a (well-written) one-liner if you want the file name, line match, and the last write time of the actual file to be returned. I recommend using an intermediary PSCustomObject for this and we can loop over the found files and matches individually:
# Use -File to only get file objects
$foundMatchesInFiles = Get-ChildItem -Recurse -File | ForEach-Object {
# Assign $PSItem/$_ to $file since we will need it in the second loop
$file = $_
# Run Select-String on each found file
$file | Select-String -Pattern CYCLE | ForEach-Object {
[PSCustomObject]#{
Path = $_.Path
Line = $_.Line
FileLastWriteTime = $file.LastWriteTime
}
}
}
Note: I used a slightly altered name of FileLastWriteTime to exemplify that this comes from the returned file and not the match provided by Select-String, but you could use LastWriteTime if you wish to retain the original property name.
Now $foundMatchesInFiles will be a collection of files which have CYCLE occurring within them, the path of the file itself (as returned by Select-String), and the last write time of the file itself as was returned by the initial Get-ChildItem.
Additional considerations
You could also use Select-Object and computed properties but IMO the above is a more concise approach when merging properties from unrelated objects together. While not a poor approach, Select-Object outputs data with a type containing the original object type name (e.g. Selected.Microsoft.PowerShell.Commands.MatchInfo). The code may work fine but can cause some confusion when others who may consume this object in the future inspect the output members. LastWriteTime, for example, belongs to FileSystemInfo, not MatchInfo. Another developer may not understand where the property came from at first if it has the MatchInfo type referenced. It is generally a better design to create a new object with the merged properties.
That said this is a minor issue which largely comes down to stylistic preference and whether this object might be consumed by others aside from you. I write modules and scripts that many other teams in my organization consume so this is a concern for me. It may not be for you. #mklement0's answer is an excellent example of how to use computed properties with Select-Object to achieve the same functional result as this answer.
Editor's note:
The gist of this question is:
* How do I add custom properties to objects output by Get-ChildItem that contain derived path information, namely the parent folder path (as Folder and name (as Foldername)?
* How do I export the resulting objects to a CSV file?
Currently have a script running that I took from StackOverflow and modified for my own use. The purpose of the script is to look at a directory, grab file names, then export them to a directory as a .csv file.
I was able to modify the script to pull through just the file name as originally it only pulled through the path, user etc. Was able to do this by adding ,Name. For some reason I can't get the script to pull through the parent folder it's in, also wanting the parents parent folder.
I believe this part of the code is what I am having most trouble with. Upon the select I was able to add ,Name but adding ,folder or ,foldername after the select doesn't pull through properly.
ForEach-Object {$_ | Add-Member -Name "Owner" -MemberType NoteProperty -Value (Get-Acl $_.FullName).Owner -PassThru} |
Sort-Object fullname |
Select FullName,CreationTime,LastWriteTime,Owner,Name,Folder,Foldername
The [System.IO.FileInfo] instances returned by Get-ChildItem for files do not have Folder or FolderName properties.
Get-ChildItem -File $HOME\Desktop | Get-Member, for instance, will show you the available properties, and will show you that the desired information can be derived from the PSPath and PSParentPath properties.
Select-Object allows hashtable-based property definitions, so-called calculated properties, which allow you to rename and/or transform properties and/or add custom properties derived from other property values by way of a script block.
Note: You can also use calculated properties with the Format-Table and Format-List cmdlets for creating output for display only (as well as with other cmdlets - see the linked help topic).
A simplified example of what you're looking for (includes output to a CSV file):
Get-ChildItem $HOME\Desktop | Select-Object Name,
#{ Name = 'Folder'; Expression = { Convert-Path $_.PSParentPath } },
#{ Name = 'Foldername'; Expression = { ($_.PSPath -split '\\')[-2] } } |
Export-Csv Out.csv -Encoding Utf8 -NoTypeInformation
Note that, alternatively, you could add Folder and FolderName properties to the input objects via Add-Member, as you did with the Owner property in your question.
Explanation:
Note that you can get more detailed information about any of the commands mentioned by running Get-Help <command-name> -Full; add -online to view the help topic in a browser; to learn more about the -split operator, run Get-Help about_split; to learn about PowerShell's help system in general, run Get-Help Get-Help -online.
Each #{ ... } construct passed to Select-Object is a hash table that defines a calculated property to attach to each output object:
The hash table must have two entries:
Name or Label, which defines the property's name; for brevity, you may use a (case-insensitive) prefix of the key name, such as just n or l.
Expression, which defines the property's value; again, a (case-insensitive) prefix of the key name works too, such as just e.
The expression can be a mere property name (a string), in case you simply want to rename an input property, but is more typically a script block ({ ... }), which is a piece of code that gets executed for each input object, and whose output becomes the value of the property being defined; inside that script block, automatic variable $_ (or $PSItem) refers to the input object at hand.
Definition of the Folder property: Convert-Path $_.PSParentPath converts the fully qualified PowerShell path that the PSParentPath property contains - which includes a prefix identifying the drive provider - to a regular filesystem path; e.g., Microsoft.PowerShell.Core\FileSystem::C:\Users\jdoe\Desktop -> C:\Users\jdoe\Desktop.
Definition of the Foldername property: ($_.PSPath -split '\\')[-2] splits the full path into components by path separator \, and then accesses the next-to-last component (-2), which is the parent folder name; e.g., C:\Users\jdoe\Desktop\file.txt -> Desktop
'\\' must be used to represent \, because -split's 1st RHS operand is a regular expression, where \ has special meaning and must therefore be doubled to be taken as a literal.
If you wanted to support / as the path separator as well for cross-platform support, you'd use ($_.PSPath -split '[\\/]')[-2].
Export-Csv exports the objects output by Select-Object to CSV Out.csv, where the input objects' property names become the header row, and the property values the data rows.
-Encoding Utf8, required in Windows PowerShell only (PowerShell (Core) 7+ now fortunately consistently uses BOM-less UTF-8), ensures that non-ASCII characters are properly encoded; by default, Export-Csv uses ASCII encoding and simply replaces non-ASCII characters, such as foreign letters, with verbatim ? characters, resulting in loss of information; note that -Encoding Utf8 in Windows PowerShell invariably creates UTF-8 files with a BOM.
-NoTypeInformation, again required in Windows PowerShell only, suppresses a line that Export-Csv by defaults adds as the first line of the output file, which contains the full type name (class name) of the input objects (e.g., #TYPE System.Management.Automation.PSCustomObject; this is meant to facilitate later reconversion into objects).
Editor's note:
The gist of this question is:
* How do I add custom properties to objects output by Get-ChildItem that contain derived path information, namely the parent folder path (as Folder and name (as Foldername)?
* How do I export the resulting objects to a CSV file?
Currently have a script running that I took from StackOverflow and modified for my own use. The purpose of the script is to look at a directory, grab file names, then export them to a directory as a .csv file.
I was able to modify the script to pull through just the file name as originally it only pulled through the path, user etc. Was able to do this by adding ,Name. For some reason I can't get the script to pull through the parent folder it's in, also wanting the parents parent folder.
I believe this part of the code is what I am having most trouble with. Upon the select I was able to add ,Name but adding ,folder or ,foldername after the select doesn't pull through properly.
ForEach-Object {$_ | Add-Member -Name "Owner" -MemberType NoteProperty -Value (Get-Acl $_.FullName).Owner -PassThru} |
Sort-Object fullname |
Select FullName,CreationTime,LastWriteTime,Owner,Name,Folder,Foldername
The [System.IO.FileInfo] instances returned by Get-ChildItem for files do not have Folder or FolderName properties.
Get-ChildItem -File $HOME\Desktop | Get-Member, for instance, will show you the available properties, and will show you that the desired information can be derived from the PSPath and PSParentPath properties.
Select-Object allows hashtable-based property definitions, so-called calculated properties, which allow you to rename and/or transform properties and/or add custom properties derived from other property values by way of a script block.
Note: You can also use calculated properties with the Format-Table and Format-List cmdlets for creating output for display only (as well as with other cmdlets - see the linked help topic).
A simplified example of what you're looking for (includes output to a CSV file):
Get-ChildItem $HOME\Desktop | Select-Object Name,
#{ Name = 'Folder'; Expression = { Convert-Path $_.PSParentPath } },
#{ Name = 'Foldername'; Expression = { ($_.PSPath -split '\\')[-2] } } |
Export-Csv Out.csv -Encoding Utf8 -NoTypeInformation
Note that, alternatively, you could add Folder and FolderName properties to the input objects via Add-Member, as you did with the Owner property in your question.
Explanation:
Note that you can get more detailed information about any of the commands mentioned by running Get-Help <command-name> -Full; add -online to view the help topic in a browser; to learn more about the -split operator, run Get-Help about_split; to learn about PowerShell's help system in general, run Get-Help Get-Help -online.
Each #{ ... } construct passed to Select-Object is a hash table that defines a calculated property to attach to each output object:
The hash table must have two entries:
Name or Label, which defines the property's name; for brevity, you may use a (case-insensitive) prefix of the key name, such as just n or l.
Expression, which defines the property's value; again, a (case-insensitive) prefix of the key name works too, such as just e.
The expression can be a mere property name (a string), in case you simply want to rename an input property, but is more typically a script block ({ ... }), which is a piece of code that gets executed for each input object, and whose output becomes the value of the property being defined; inside that script block, automatic variable $_ (or $PSItem) refers to the input object at hand.
Definition of the Folder property: Convert-Path $_.PSParentPath converts the fully qualified PowerShell path that the PSParentPath property contains - which includes a prefix identifying the drive provider - to a regular filesystem path; e.g., Microsoft.PowerShell.Core\FileSystem::C:\Users\jdoe\Desktop -> C:\Users\jdoe\Desktop.
Definition of the Foldername property: ($_.PSPath -split '\\')[-2] splits the full path into components by path separator \, and then accesses the next-to-last component (-2), which is the parent folder name; e.g., C:\Users\jdoe\Desktop\file.txt -> Desktop
'\\' must be used to represent \, because -split's 1st RHS operand is a regular expression, where \ has special meaning and must therefore be doubled to be taken as a literal.
If you wanted to support / as the path separator as well for cross-platform support, you'd use ($_.PSPath -split '[\\/]')[-2].
Export-Csv exports the objects output by Select-Object to CSV Out.csv, where the input objects' property names become the header row, and the property values the data rows.
-Encoding Utf8, required in Windows PowerShell only (PowerShell (Core) 7+ now fortunately consistently uses BOM-less UTF-8), ensures that non-ASCII characters are properly encoded; by default, Export-Csv uses ASCII encoding and simply replaces non-ASCII characters, such as foreign letters, with verbatim ? characters, resulting in loss of information; note that -Encoding Utf8 in Windows PowerShell invariably creates UTF-8 files with a BOM.
-NoTypeInformation, again required in Windows PowerShell only, suppresses a line that Export-Csv by defaults adds as the first line of the output file, which contains the full type name (class name) of the input objects (e.g., #TYPE System.Management.Automation.PSCustomObject; this is meant to facilitate later reconversion into objects).