I am trying to create a very specific Powershell report script that I can input a server list via CSV or TXT, and have it check a specific folder who's path exists on every server. Then output that information to a formatted CSV file with the server name, and the total size of that single folder tree.
I've been looking at the following thread: (Powershell folder size of folders without listing Subdirectories, as it seems to have the closest code examples of what I need to accomplish, but none of them are setup for a server report input and output.
The script shouldn't need any credentials as it will be running with elevated privileges. I don't think a function will be necessary as I plan to change the defined path as different situations come up.
I'm usually pretty good at piecing together different scripts I find. However I'm a bit stumped trying to find something that matches this exact function, so I'm not even sure where to start. I'm new to powershell and not very good at writing complex scripts by hand yet.
servers.txt
server1
server2
there are two ways, one by smb, one using winRM
1 + Powershell folder size of folders without listing Subdirectories
$servers = Get-Content "$PSScriptRoot\servers.txt"
$path = "c$\Distr"
$result = #()
foreach ($server in $servers){
$fullpath = "\\" + $server + "\" + "$path"
$fullpath
$size = (Get-ChildItem -Recurse -Force $fullpath | Measure-Object -Property Length -Sum).sum
$array = "" | Select server, size
$array.server = $server
$array.size = $size
$result+=$array
}
$result |Export-Csv -Delimiter ";" -Encoding UTF8 -Path "$PSScriptRoot\result.csv"
if the network connection is slow enough, it's better to use WinRM, but I don't have an example at hand right now.
Thanks to rinat gadev for the starter code. Here's my final with adjustments that does exactly what I wanted.
$servers = Get-Content "$PSScriptRoot\servers.txt"
$path = "c$\Distr"
$result = #()
foreach ($server in $servers){
$fullpath = "\\" + $server + "\" + "$path"
$fullpath
$size = (Get-ChildItem -Recurse -Force $fullpath | Measure-Object -Property Length -Sum ).sum | ForEach-Object {[math]::Round($_/1GB,2)}
$array = "" | Select-Object Server, Size
$array.server = $server
$array.size = $size
$result+=$array
}
$result | Export-Csv -Path "$PSScriptRoot\result.csv"-NoTypeInformation -UseCulture
I need to delete all the archived files and folder older than 15 days.
I have implemented the solution using PowerShell script but it taking more than a day to delete all files. Total size of the folder is less than 100 GB.
$StartFolder = "\\Guru\Archive\"
$deletefilesolderthan = "15"
#Get Foldernames for ForEach Loop
$SubFolders = Get-ChildItem -Path $StartFolder |
Where-Object {$_.PSIsContainer -eq "True"} |
Select-Object Name
#Loop through folders
foreach ($Subfolder in $SubFolders) {
Write-Host "Processing Folder:" $Subfolder
#For each folder recurse and delete files olders than specified number of days while the folder structure is left intact.
Get-ChildItem -Path $StartFolder$($Subfolder.name) -Include *.* -File -Recurse |
Where LastWriteTime -lt (Get-Date).AddDays(-$deletefilesolderthan) |
foreach {$_.Delete()}
#$dirs will be an array of empty directories returned after filtering and loop until until $dirs is empty while excluding "Inbound" and "Outbound" folders.
do {
$dirs = gci $StartFolder$($Subfolder.name) -Exclude Inbound,Outbound -Directory -Recurse |
Where {(gci $_.FullName).Count -eq 0} |
select -ExpandProperty FullName
$dirs | ForEach-Object {Remove-Item $_}
} while ($dirs.Count -gt 0)
}
Write-Host "Completed" -ForegroundColor Green
#Read-Host -Prompt "Press Enter to exit"
Please suggest some way to optimise the performance.
If you have many smaller files, the long delete time is not abnormal because it has to process each file descriptor. Some improvements can be made depending on your version; I'm going to assume you're on at least v4.
#requires -Version 4
param(
[string]
$start = '\\Guru\Archive',
[int]
$thresholdDays = 15
)
# getting the name wasn't useful. keep objects as objects
foreach ($folder in Get-ChildItem -Path $start -Directory) {
"Processing Folder: $folder"
# get all items once
$folders, $files = ($folder | Get-ChildItem -Recurse).
Where({ $_.PSIsContainer }, 'Split')
# process files
$files.Where{
$_.LastWriteTime -lt (Get-Date).AddDays(-$thresholdDays)
} | Remove-Item -Force
# process folders
$folders.Where{
$_.Name -notin 'Inbound', 'Outbound' -and
($_ | Get-ChildItem).Count -eq 0
} | Remove-Item -Force
}
"Complete!"
The reason why it takes so many time is that you are deleting files/folder over network which leads to need for additional network communication for every file and folder. You can easily check that fact using network analyzer. The best approach here is to use one of the method that allows to run code which executes file operations on remote machine, for example you can try to use:
WinRM
psexec (first copy code to remote machine and then execute it using psexec)
remote WMI (using CIM_Datafile)
or even adding needed task to the scheduler
I would prefer to use WinRM but psexec is also good decision (if you don't want to perform additional configuration of WinRM).
Is it possible to use $SecretFolder from the else statement in future Iterations if the company is the same. E.g. Multiple users exist on the list from one company but they all need to have a link generated for 1 folder for the company to access.
#Location of original dataset
$csv = Import-Csv c:\export.csv
#loops through every line of the csv
Foreach ($line in $csv){
#Generate random folder name (8 Characters long)
$SecretFolder = -join ((48..57) + (97..122) | Get-Random -Count 8 | % {[char]$_})
#Create URL
$url = "www.website.com.au/2017Rates/$SecretFolder"
#Test: Has the company already had a folder created
if (Get-Variable $line.CompanyName -Scope Global -ErrorAction SilentlyContinue)
{
#Append URL to CSV for a person who already has a company folder
$report =#()
$report += New-Object psobject -Property #{CompanyName=$line.CompanyName;FirstName=$line.FirstName;LastName=$line.LastName;EmailAddress=$line.EmailAddress;'Letter Type'=$line.'Letter Type';URL=$URL}
$report | export-csv testreporting.csv -Append
}
else
{
#Create Folder with Random Cryptic name
mkdir C:\Users\bford\test\$SecretFolder
#Copy item from FileLocation in CSV to SecretFolder Location
Copy-Item -Path $line.FileLocation -Destination c:\users\bford\test\$SecretFolder -Recurse -ErrorAction SilentlyContinue
#Create Variable for Logic test with the Name CompanyName
New-Variable -Name $line.CompanyName
#Append csv with the updated details
$S_report =#()
$S_report += New-Object psobject -Property #{CompanyName=$line.CompanyName;FirstName=$line.FirstName;LastName=$line.LastName;EmailAddress=$line.EmailAddress;'Letter Type'=$line.'Letter Type';URL=$url}
$S_report | export-csv testreporting.csv -Append
}
}
#Cleanup remove all the variables added
Remove-Variable * -ErrorAction SilentlyContinue
Do you have any reason to think it's impossible? Yeah it's possible, you should Google hashtables and find that they do everything you're trying to do with get-variable, only way better.
But your question amounts to "how do I rewrite my script so it works?" and rewriting your script to me means getting rid of the duplicate #()+= triple lines, the mystery numbers, the global variables, and the extra variables and the if/else, and it ends up a completely different script altogether.
A completely different, and mostly untested, script:
# import and group all people in the same company together
# then loop over the groups (companies)
Import-Csv -Path c:\export.csv |
Group-Object -Property CompanyName |
ForEach-Object {
# This loop is once per company, make one secret folder for this company.
$SecretFolder = -join ( [char[]]'abcdefghijklmnopqrstuvwxyz1234567890' | Get-Random -Count 8 )
New-Item -ItemType Directory -Path "C:\Users\bford\test\$SecretFolder"
# Loop through all the people in this company, and copy their files into this company's secret folder
$_.Group | ForEach-Object {
Copy-Item -Path $_.FileLocation -Destination c:\users\bford\test\$SecretFolder -Recurse -ErrorAction SilentlyContinue
}
# Output each person in this company with just the properties needed, and a new one for this company's URL
$_.Group | Select-Object -Property CompanyName , FirstName,
LastName, EmailAddress, 'Letter Type',
#{Name='Url'; Expression={"www.website.com.au/2017Rates/$SecretFolder"}}
} | Export-Csv -Path testreporting.csv -NoTypeInformation
But to edit your script to do what you want, use a hashtable, e.g.
$SecretFolders = #{} #at top of your script, outside loops
# in loops:
if (-not $SecretFolders.ContainsKey($line.CompanyName))
{
$SecretFolders[$line.CompanyName] = -join (random name generation here)
}
$SecretFolder = $SecretFolders[$line.CompanyName]
My script giving error "Get-ChildItem : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be
less than 248 characters." and "Measure-Object : The property "length" cannot be found in the input for any objects".
It supposed to create a text file with list of folders arranged by size. However,I get errors above and some folders get size 0.0 Mb. Do you have any suggestions?
$invocation = (Get-Variable MyInvocation).Value
$directorypath = Split-Path $invocation.MyCommand.Path
Function Get-FolderName
{
Param (
$PathName
)
$Array=#('\')
ForEach ($Folder in $PathName)
{
$a=[string]$Folder
$a=$a.split("\")[-1]
$a=$a-replace '["}"]',''
$Array+=$a
}
ForEach ($Folder in $Array)
{
$colItems = (Get-ChildItem $directorypath\$Folder -recurse -ErrorAction Inquire | Measure-Object -property length -sum -ErrorAction Inquire )
$colItems ="{0:N2}" -f ($colItems.sum / 1MB)
$colItems =[String]$colItems-replace '[","]',''
#$colItems =[String]$colItems-replace '["."]',''
$colItems = [float]$colItems
$colItemsGB = [float]($colItems /1024)| % { '{0:0.##}' -f $_ }
#$colItems=(Get-ChildItem $directorypath\$Folder -Recurse | Measure-Object Length -Sum)
[PSCustomObject]#{
Folder = $Folder;
Size_MB=$colItems
Size_GB=$colItemsGB
}
}
}
Function Check-FileName{
Param (
$PathName
)
$b=[string]$directorypath.split("\")[-1]
if(Test-Path $directorypath\$b.txt)
{
Clear-Content $directorypath\$b.txt
$content = Get-FolderName $PathName
return $content|Sort-Object -property Size_MB -Descending | out-file $b".txt"
}
else {
$content = Get-FolderName $PathName
return $content |Sort-Object -property Size_MB -Descending| out-file $b".txt"
}
}
$aa = dir -Directory | Select FullName
Check-FileName $aa
There is almost nothing you can do about it. Situation is application specific. Windows API for accessing longer file paths exists but its not certain if application developers used it. For instance, TFS server 2015 can't use longer file paths also and there is no fix that I am aware of. This is typically problem with Java apps since they can have very long class paths.
You could use AlphaFS.
Other alternative includes using New-PSDrive which involves some work to join the results later.
I am fairly new to PowerShell, and wrote a script to analyze network shares, dump it to CSV and import it to SQL. Our NAS device has several hidden shares, so I specify the (NAS) server name, a string of share names to search, and the folder depth that I want to search. (like 3 or 4 levels for quick testing).
The script tries to convert the security permissions to show simple "List, Read or Modify" access to folders. Can this user/group "list" the files, view the files, or modify them? The user info is put into a comma-separated list for each access type.
I suspect that although the code is functional, it may not be very efficient and I wonder if there are some significant improvements that could be made?
To deal with long pathnames, I use the "File System Security PowerShell Module 3.2.3" which appends a "2" to several modules, like "Get-ChildItem2".
I used to just specify one share folder, and I'm also wondering if my For-Each-Object that processes multiple shares has introduced a bug in how the objects are handled. It seems to use a lot more memory and slows down, and doesn't seem to process the last share in the list properly.
Here is the code: (split into 3 pieces)
# This script reads through the specified shares on the server and creates a CSV file containing the folder information
# The data is written to a SQL server
$Server = '\\MyServer'
$Shares = 'data$,share$'.Split(',')
$Levels = 99 # specify 3 or 4 for faster testing with less info
$ScanDate = Get-Date -Format 'yyyy-MM-dd HH:mm:ss'
$CSVFile = 'C:\FolderInfo\' + $ScanDate.Replace(':','-') + '.csv'
Write-Debug "ScanDate will be set to: $ScanDate"
Write-Debug "Data will be written to: $CSVFile"
$Separator = ',' # Separate the AD groups
$ListRights = 'ListDirectory,GenericExecute'.Split(',')
$ReadRights = 'Read,ReadAndExecute,GenericRead'.Split(',')
$ModifyRights = 'CreateFiles,AppendData,Write,Modify,FullControl,GenericAll'.Split(',')
$ErrorPref = 'Continue'
$ErrorActionPreference = $ErrorPref
$DebugPreference = 'Continue'
$DataBase = 'Folders.dbo.FolderInfo'
Function Get-Subs {
Param([String]$Path,[Byte]$Depth)
$CurrentDepth = $Path.Length - $Path.Replace('\','').Length
new-object psobject -property #{Path=$Path} # Object 'Path' is for the pipe output
If ( $CurrentDepth -lt ($Depth + 1) ) {
Get-ChildItem2 -Path $Path -Directory | ForEach {
Get-Subs $PSItem.FullName $Depth }
}
}
The next line has a commented out line of code that I was using to test how it is processing multiple share names, and it works properly, but the remaining code below seems to mess up on the last sharename in the list.
$Shares | ForEach-Object {Get-Subs (Resolve-Path $Server\$_).ProviderPath $Levels} | Get-Item2 | #ForEach-Object { new-object psobject -property #{Path=$_.FullName} } | Select Path
And the remaining code: (I hope this breakage doesn't confuse everyone :)
ForEach-Object {
$ListUsers = #()
$ReadUsers = #()
$ModifyUsers = #()
$Folder = $PSItem.FullName
Write-Debug $Folder
$Inherited = $true
try {$Owner = (Get-NTFSOwner -Path $Folder).Owner.AccountName.Replace('MyDomain\','')
}
catch {Write-Debug "Access denied: $Folder"
$Owner = 'access denied'
$Inherited = $false
}
$Levels = $Folder.Length - $Folder.Replace('\','').Length - 3 # Assuming \\server\share as base = 0
Get-NTFSAccess $Folder | Where { $PSItem.Account -ne 'BUILTIN\Administrators' } | ForEach-Object {
$Account = $PSItem.Account.AccountName.Replace('MyDomain\','')
$Rights = $PSItem.AccessRights -split(',')
If ($PSItem.IsInherited -eq $false) {$Inherited = $false}
IF ($PSItem.InheritanceFlags -eq 'ContainerInherit') { # Folders only or 'ContainerInherit, ObjectInherit' = Folders and Files
If (#(Compare -ExcludeDifferent -IncludeEqual ($ListRights)($Rights)).Length -and $Account) {$ListUsers += $Account}
If (#(Compare -ExcludeDifferent -IncludeEqual ($ReadRights)($Rights)).Length -and $Account) {$ListUsers += $Account}
If (#(Compare -ExcludeDifferent -IncludeEqual ($ModifyRights)($Rights)).Length -and $Account) {$ListUsers += $Account
Write-Debug "Modify anomaly found on Container only: $Account with $Rights in $Folder"
}
}
Else {
If (#(Compare -ExcludeDifferent -IncludeEqual ($ListRights)($Rights)).Length -and $Account) {$ListUsers += $Account}
If (#(Compare -ExcludeDifferent -IncludeEqual ($ReadRights)($Rights)).Length -and $Account) {$ReadUsers += $Account}
If (#(Compare -ExcludeDifferent -IncludeEqual ($ModifyRights)($Rights)).Length -and $Account) {$ModifyUsers += $Account}
}
}
$FileCount = Get-ChildItem2 -Path $Folder -File -IncludeHidden -IncludeSystem | Measure-Object -property Length -Sum
If ($FileCount.Sum) {$Size = $FileCount.Sum} else {$Size = 0}
If ($FileCount.Count) {$NumFiles = $FileCount.Count} else {$NumFiles = 0}
$ErrorActionPreference = 'SilentlyContinue'
Remove-Variable FolderInfo
Remove-Variable Created, LastAccessed, LastModified
$ErrorActionPreference = $ErrorPref
$FolderInfo = #{} # create empty hashtable, new properties will be auto-created
$LastModified = Get-ChildItem2 -Path $Folder -File | Measure-Object -property LastWriteTime -Maximum
IF ($LastModified.Maximum) {$FolderInfo.LastModified = $LastModified.Maximum.ToString('yyyy-MM-dd hh:mm:ss tt')}
else {$FolderInfo.LastModified = $PSItem.LastWriteTime.ToString('yyyy-MM-dd hh:mm:ss tt')}
$LastAccessed = Get-ChildItem2 -Path $Folder -File | Measure-Object -property LastAccessTime -Maximum
IF ($LastAccessed.Maximum) {$FolderInfo.LastAccessed = $LastAccessed.Maximum.ToString('yyyy-MM-dd hh:mm:ss tt')}
else {$FolderInfo.LastAccessed = $PSItem.LastAccessTime.ToString('yyyy-MM-dd hh:mm:ss tt')}
$Created = Get-ChildItem2 -Path $Folder -File | Measure-Object -Property CreationTime -Maximum
IF ($Created.Maximum) {$FolderInfo.Created = $Created.Maximum.ToString('yyyy-MM-dd hh:mm:ss tt')}
else {$FolderInfo.Created = $PSItem.CreationTime.ToString('yyyy-MM-dd hh:mm:ss tt')}
$FolderInfo.FolderName = $Folder
$FolderInfo.Levels = $Levels
$FolderInfo.Owner = $Owner
$FolderInfo.ListUsers = $ListUsers -join $Separator
$FolderInfo.ReadUsers = $ReadUsers -join $Separator
$FolderInfo.ModifyUsers = $ModifyUsers -join $Separator
$FolderInfo.Inherited = $InheritedFrom
$FolderInfo.Size = $Size
$FolderInfo.NumFiles = $NumFiles
$FolderInfo.ScanDate = $ScanDate
Write-Debug $Folder
Write-Output (New-Object –Typename PSObject –Prop $FolderInfo)
} | Select FolderName, Levels, Owner, ListUsers, ReadUsers, ModifyUsers, Inherited, Size, NumFiles, Created, LastModified, LastAccessed, ScanDate |
ConvertTo-csv -NoTypeInformation -Delimiter '|' |
ForEach-Object {$PSItem.Replace('"','')} |
Out-File -FilePath $CSVFile -Force
Write-debug 'Starting import...'
$Query = #"
BULK INSERT $DataBase FROM '$CSVFile' WITH (DATAFILETYPE = 'widechar', FIRSTROW = 2, FIELDTERMINATOR = '|', ROWTERMINATOR = '\n')
"#
sqlcmd -S MyComputer\SQLExpress -E -Q $Query
Arrays can be defined by comma separated list. Each element belongs in quotes.
For $Shares = 'data$,share$'.Split(','), try, $Shares = 'data$','share$'
Similarly for most of your arrays where you used .Split(',')
The filename can be done in many, many ways. Here you use a custom format then change it immediately. Recommend you replace
$ScanDate = Get-Date -Format 'yyyy-MM-dd HH:mm:ss'
$CSVFile = 'C:\FolderInfo\' + $ScanDate.Replace(':','-') + '.csv'
With
$ScanDate = Get-Date -Format 'yyyy-MM-dd-HH-mm-ss'
$CSVFile = "C:\FolderInfo\$ScanDate.csv"
This uses custom date format to set to what you wanted without the extra operation AND leverages PS way of evaluating variables within strings if the string is in double quotes. YMMV, but I also prefer 'yyyyMMdd-HHmmss' for datestamps.
Why are you defining a variable only to use it to define a second variable?
$ErrorPref = 'Continue'
$ErrorActionPreference = $ErrorPref
Why not $ErrorActionPreference = 'Continue'?
I found it later. Could probably use an explanation of what you're doing and how to do it when you define your preference. e.g. # Set default errorAction to 'Continue' during development, to show errors for debugging. Change this to 'silentlycontinue' to ignore errors during run. This will really help when you come back to this script in 18 months and are like WTF does this do?
Also, research Advanced Functions and CmdletBinding() so that you can build your function like a commandlet, including inputting a -debug switch, so you can write with debugging in mind.
What does Function Get-Subs actually do? It looks like some kind of recursion to get the path using the custom commandlet get-childitem2. Do you need the full path? Get-ChildItem $path -Directory -Recurse | select fullname where path is your UNC path or local path or any other provider, really.
Get-NTFSOwner not sure where this comes from, perhaps your custom module. You can use Get-ACL in Powershell 3 (not sure about 2, I don't remember). $owner = (Get-ACL 'path\file.ext').Owner.Replace('mydomain\','')
No idea what you're doing with all the inheritance stuff. Just keep in mind that you can get paths from Get-ChildItem | select Fullname and owner from Get-ACL. These may allow you to skip the custom module.
For:
$FileCount = Get-ChildItem2 -Path $Folder -File -IncludeHidden -IncludeSystem | Measure-Object -property Length -Sum
If ($FileCount.Sum) {$Size = $FileCount.Sum} else {$Size = 0}
If ($FileCount.Count) {$NumFiles = $FileCount.Count} else {$NumFiles = 0}
Use:
$files = Get-ChildItem -Force -File -Path $folder
-Force shows all files. -File limits it to files only. Then the number of files is $files.count. Works with empty folders, folders with one hidden file, and files with hidden and normal files.
For $folderinfo consider using a custom object. If you create it within the loop it should destroy the previous one. Then you can assign values directly to the object instead of storing them in a variable then inserting the variable into the hash table.
Using Get-ChildItem native will help you maintain this script far more easily than your customized module.
For:
| Select FolderName, Levels, Owner, ListUsers, ReadUsers, ModifyUsers, Inherited, Size, NumFiles, Created, LastModified, LastAccessed, ScanDate |
ConvertTo-csv -NoTypeInformation -Delimiter '|' |
ForEach-Object {$PSItem.Replace('"','')} |
Out-File -FilePath $CSVFile -Force
Try:
| Export-CSV -NoTypeInformation $CSVFile # by default this will overwrite, but you're timestamping down to the second so I don't think this will be an issue.
Overall:
Get rid of the custom module, I think everything you're doing here can be done in native PowerShell.
Consider recursion
Use an advanced function, complete with documentation. Ref: https://technet.microsoft.com/en-us/magazine/hh360993.aspx
Definitely use a custom object to store data for each file/folder. Passing a collection of objects to Export-CSV produces excellent output.
pipe to Select -ExpandProperty when an object contains a hashtable or another object, e.g. Get-ACL 'file.txt' | select -ExpandProperty Access gives a list of the access rule objects in the ACL.
Get-Help and Get-Member are amongst the most powerful commands in PowerShell.
Select-Object and Where-Object are up there too.