Optimizing VERY Slow PS Script - powershell

I have a PS script that is taking CSV files (the largest being about 170MB) and splitting it into multiple smaller CSVs based on the ListGuid value. It is then taking each file and uploading it to a specific path in SharePoint using PnP based on the Web Guid and List Guid. This script it taking forever to run and I am having trouble finding ways to optimize it. Any help would be appreciated. Here is the script:
$PermissionsFile = Get-ChildItem -Path $downloadFilePath -Filter *.csv
foreach ($file in $PermissionsFile) {
$SiteCollectionReport = Import-Csv -Path "$downloadFilePath/$($file.Name)"
$filteredListTestFile = $SiteCollectionReport | Where-Object {$_.Type -eq "List"}
$groupedListFile = $filteredListTestFile | Select-Object Url, ListGuid -Unique
$subWebsConnection = Connect-SharePoint -WebUrl $SiteCollectionReport.Url[0] -CheckForAppCredentials
$subWebs = Get-PnPSubWebs -Recurse -IncludeRootWeb -Connection $subWebsConnection | Select-Object Url, Id
$permissionsSiteConnection = Connect-SharePoint -WebUrl "https://company.sharepoint.com/sites/edmsc/Internal" -CheckForAppCredentials
foreach ($guid in $groupedListFile) {
$webGuid
$listGuid = $guid.ListGuid
$SiteCollectionReport | Where-Object {$_.ListGuid -like $listGuid -and $_.Type -eq "List"} | Export-Csv -Path "Path\Permissions $($listGuid).csv" -NoTypeInformation
$url = $guid.Url
$siteCollectionName = $url.Split("/")[4]
if ($url.Contains(" ")) {
$url = $url.Replace(" ","%20")
}
$split = $url.substring(0, $url.LastIndexOf("/"))
if ($split.Contains("Lists")) {
$split = $split -split "Lists"
}
foreach ($web in $subWebs) {
if ($web.Url -eq $split) {
$webGuid = $web.Id
#Write-Host "Adding permissions reports for $split"
#Write-Host "List Guid $listGuid"
Write-Host "Web Guid $webGuid"
}
}
$fieldValues = #{"ObjectType"="List/ListItem"; "WebGuid"=$webGuid; "ListGuid"=$listGuid}
#$permissionsSiteWeb = Get-PnPWeb -Connection $permissionsSiteConnection
Add-PnPFile -Path "Path\Permissions $($listGuid).csv" -Folder "SiteCollectionPermissions/$siteCollectionName/$webGuid" -Values $fieldValues -Connection $permissionsSiteConnection
}
Write-Host "Deleting Permissions Files..."
Get-ChildItem -Path "Path" -Include *.csv* -File -Recurse | ForEach-Object { $_.Delete()}
}

Related

command 'search-index' does not appear to be working

here is the script I've created. I have downloaded the 'PSSearch' package and when I goto commands 'Search-Index' is one of the available commands
$computers = #([some computer])
$destination = "[some path]"
foreach ($computer in $computers) {
$Path = Set-Location [path on computer]
$keywords= #('"word 1"','word2','word3','word4')
$dirlist = Get-ChildItem -Recurse -Force $Path -ErrorAction Continue
foreach($word in $keywords) {
$SearchResults = Search-Index $word
$dirlist | Where-Object {$_.Name -match $SearchResults} | Select-Object Name,FullName | format-Table * -AutoSize |
Export-Csv $destination\FoundFiles.csv -nti -Append
$cui = ($dirlist | Where-Object {$_.Name -match $SearchResults})
Copy-Item $cui -Destination $destination - Append
}
}
What is happening is I'm getting all files and folders from the location (not just the ones I'm searching for)
The problem could be that I don't know how this line should be scripted
$cui = ($dirlist | Where-Object {$_.Name -match $SearchResults})

Powershell Filter for mathing AD Account to Folder

I am trying to match the samaccountname to the current folder patch, and it would worked fine if the folderpatch was "d:\profile\username" but instead of username its states username_S-1-5-21*....
I use the below code, but is it possible to filter everything behind the _S-1.... so it can match the username to the samaccountname ?
I use the below code, any help would be appreciated
[EDIT: added complete script]
Below is the complete script, so the issue is that i have several user folder with _S-1-5-21* behind the username in the folder of our FXLogic profile folder and need to match the samaccountname with the folder ( username - _S-1-5-21* )
I hope this explanation is more clear, and yes its a SID not a GUID always get them mixed up.
param(
[Parameter(Mandatory=$true)]
$FXLogicFolderPath,
$MoveFolderPath,
$SearchBase,
[string[]]$ExcludePath,
[switch]$FolderSize,
[switch]$MoveDisabled,
[switch]$DisplayAll,
[switch]$UseRobocopy,
[switch]$RegExExclude,
[switch]$CheckFXLogicDirectory)
Check if FXLogicFolderPath is found, exit with warning message if path is incorrect
if (!(Test-Path -LiteralPath $FXLogicFolderPath)){
Write-Warning "FXLogicFolderPath not found: $FXLogicFolderPath"
Check if MoveFolderPath is found, exit with warning message if path is incorrect
if ($MoveFolderPath) {
if (!(Test-Path -LiteralPath $MoveFolderPath)){
Write-Warning "MoveFolderPath not found: $MoveFolderPath"
exit
}}
exit
Main loop, for each folder found under FXLogic folder path AD is queried to find a matching samaccountname
$ListOfFolders = Get-ChildItem -LiteralPath "$FXLogicFolderPath" -Force | Where-Object {$_.PSIsContainer}
Exclude folders if the ExcludePath parameter is given
if ($ExcludePath) {
$ExcludePath | ForEach-Object {
$CurrentExcludePath = $_
if ($RegExExclude) {
$ListOfFolders = $ListOfFolders | Where-Object {$_.FullName -notmatch $CurrentExcludePath}
} else {
$ListOfFolders = $ListOfFolders | Where-Object {$_.FullName -ne $CurrentExcludePath}
}
}}
$ListOfFolders | ForEach-Object {
$CurrentPath = Split-Path -Path $_ -Leaf
Construct AD Searcher, add SearchRoot attribute if SearchBase parameter is specified
$ADSearcher = New-Object DirectoryServices.DirectorySearcher -Property #{
Filter = "(samaccountname=$CurrentPath)"
}
if ($SearchBase) {
$ADSearcher.SearchRoot = [adsi]$SearchBase
}
Use the FullName path to look for a FXLogicdirectory attribute and replace the backslash by the \5C LDAP escape character
if ($CheckFXLogicDirectory) {
$ADSearcher.Filter = "(FXLogicdirectory=$($_.FullName -replace '\\','\5C')*)"
}
Execute AD Query and store in $ADResult
$ADResult = $ADSearcher.Findone()
If no matching samaccountname is found this code is executed and displayed
if (!($ADResult)) {
$HashProps = #{
'Error' = 'Account does not exist and has a FXLogic folder'
'FullPath' = $_.FullName
}
if ($FolderSize) {
$HashProps.SizeinBytes = [long](Get-ChildItem -LiteralPath $_.Fullname -Recurse -Force -ErrorAction SilentlyContinue |
Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue | Select-Object -Exp Sum)
$HashProps.SizeinMegaBytes = "{0:n2}" -f ($HashProps.SizeinBytes/1MB)
}
if ($MoveFolderPath) {
$HashProps.DestinationFullPath = Join-Path -Path $MoveFolderPath -ChildPath (Split-Path -Path $_.FullName -Leaf)
if ($UseRobocopy) {
robocopy $($HashProps.FullPath) $($HashProps.DestinationFullPath) /E /MOVE /R:2 /W:1 /XJD /XJF | Out-Null
} else {
Move-Item -LiteralPath $HashProps.FullPath -Destination $HashProps.DestinationFullPath -Force
}
}
Output the object
New-Object -TypeName PSCustomObject -Property $HashProps
If samaccountname is found but the account is disabled this information is displayed
} elseif (([boolean]((-join $ADResult.Properties.useraccountcontrol) -band 2))) {
$HashProps = #{
'Error' = 'Account is disabled and has a FXLogic folder'
'FullPath' = $_.FullName
}
if ($FolderSize) {
$HashProps.SizeinBytes = [long](Get-ChildItem -LiteralPath $_.Fullname -Recurse -Force -ErrorAction SilentlyContinue |
Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue | Select-Object -Exp Sum)
$HashProps.SizeinMegaBytes = "{0:n2}" -f ($HashProps.SizeinBytes/1MB)
}
if ($MoveFolderPath -and $MoveDisabled) {
$HashProps.DestinationFullPath = Join-Path -Path $MoveFolderPath -ChildPath (Split-Path -Path $_.FullName -Leaf)
Move-Item -LiteralPath $HashProps.FullPath -Destination $HashProps.DestinationFullPath -Force
}
Output the object
New-Object -TypeName PSCustomObject -Property $HashProps
Folders that do have active user accounts are displayed if -DisplayAll switch is set
} elseif ($ADResult -and $DisplayAll) {
$HashProps = #{
'Error' = $null
'FullPath' = $_.FullName
}
if ($FolderSize) {
$HashProps.SizeinBytes = [long](Get-ChildItem -LiteralPath $_.Fullname -Recurse -Force -ErrorAction SilentlyContinue |
Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue | Select-Object -Exp Sum)
$HashProps.SizeinMegaBytes = "{0:n2}" -f ($HashProps.SizeinBytes/1MB)
}
Output the object
New-Object -TypeName PSCustomObject -Property $HashProps
}}
# Construct AD Searcher, add SearchRoot attribute if SearchBase parameter is specified
$ADSearcher = New-Object DirectoryServices.DirectorySearcher -Property #{
Filter = "(samaccountname=$CurrentPath)"
}
if ($SearchBase) {
$ADSearcher.SearchRoot = [adsi]$SearchBase
The S-1-5-21... part is not a GUID, it's a SID (the principal's Security Identifier).
You can use the -replace operator to remove that part of the folder name:
$folderName = 'username_S-1-5-21-2855571654-3033049851-1520320983-9328'
$userName = $folderName -replace '_S-1-5.*$'
After which you can construct the desired LDAP query filter:
$ADSearcher = New-Object DirectoryServices.DirectorySearcher -Property #{
Filter = "(samaccountname=$userName)"
}
Not sure where your username is being derived from but you could do something simple with a regex replace on the $CurrentPath
$currentPath = 'username_S-1-5-21-2855571654-3033049851-1520320983-9328'
$currentPath = $currentPath -replace '_.+'
# AD Searcher here
This will replace everything after the underscore but all usernames will have to be in a consistent format or you'll have to take into account all possible formats that could be encountered.

Shortcut List script

I've created a script that will read a list of usernames in a file named Users.csv and then search a network share for their user profile. It then reports on the shortcut target paths of said user profile and exports that to another .csv.
It works great when I use it per user but when I get it to report the data to the .csv file I can only seem to get it to report on the last user in my Users.csv and not each one. I was thinking it is because the export part of the script overwrites the report.csv for each user it runs and I need it to create a unique report.csv for each username. Anyone have any ideas?
$Users = (Get-Content C:\temp\Users.csv) -notmatch '^\s*$'
foreach ($User in $Users) {
$Shortcuts = Get-ChildItem -Recurse \\UNCPATHTOUSERPROFILES\$User -Include *.lnk
$Shell = New-Object -ComObject WScript.Shell
$data = foreach ($Shortcut in $Shortcuts) {
[PSCustomObject]#{
ShortcutName = $Shortcut.Name;
Target = $Shell.CreateShortcut($Shortcut).targetpath
User = $User
}
}
[Runtime.InteropServices.Marshal]::ReleaseComObject($Shell) | Out-Null
}
foreach ($User in $Users) {
$data | Export-Csv c:\temp\report.csv -NoTypeInformation
}
You should combine your two outer foreach loops, not only since they're iterating the same collection, but because the second loop is trying to use a variable, $data, created in the first loop. Further, since Export-Csv is being called in a loop, you will need to pass the -Append parameter to prevent the output file from being overwritten each time.
$Users = (Get-Content C:\temp\Users.csv) -notmatch '^\s*$'
foreach ($User in $Users) {
$Shortcuts = Get-ChildItem -Recurse \\UNCPATHTOUSERPROFILES\$User -Include *.lnk
$Shell = New-Object -ComObject WScript.Shell
$data = foreach ($Shortcut in $Shortcuts) {
[PSCustomObject]#{
ShortcutName = $Shortcut.Name;
Target = $Shell.CreateShortcut($Shortcut).targetpath
User = $User
}
}
$data | Export-Csv c:\temp\report.csv -Append -NoTypeInformation
[Runtime.InteropServices.Marshal]::ReleaseComObject($Shell) | Out-Null
}
You could also eliminate the $Shortcuts and $data variables in favor of using the pipeline...
$Users = (Get-Content C:\temp\Users.csv) -notmatch '^\s*$'
foreach ($User in $Users) {
$Shell = New-Object -ComObject WScript.Shell
Get-ChildItem -Recurse \\UNCPATHTOUSERPROFILES\$User -Include *.lnk `
| ForEach-Object -Process {
[PSCustomObject]#{
ShortcutName = $_.Name;
Target = $Shell.CreateShortcut($_).targetpath
User = $User
}
} `
| Export-Csv c:\temp\report.csv -Append -NoTypeInformation
[Runtime.InteropServices.Marshal]::ReleaseComObject($Shell) | Out-Null
}
...but note that -Append is still required. Finally, you could rewrite the whole thing using the pipeline...
$Shell = New-Object -ComObject WScript.Shell
try
{
Get-Content C:\temp\Users.csv `
| Where-Object { $_ -notmatch '^\s*$' } -PipelineVariable 'User' `
| ForEach-Object -Process { "\\UNCPATHTOUSERPROFILES\$User" } `
| Get-ChildItem -Recurse -Include *.lnk `
| ForEach-Object -Process {
[PSCustomObject]#{
ShortcutName = $_.Name;
Target = $Shell.CreateShortcut($_).targetpath
User = $User
} `
} `
| Export-Csv c:\temp\report.csv -NoTypeInformation
}
finally
{
[Runtime.InteropServices.Marshal]::ReleaseComObject($Shell) | Out-Null
}
...so that report.csv is only opened for writing once, with no -Append needed. Note that I am creating a single WScript.Shell instance and using try/finally to ensure it gets released.
In the event a user in Users.csv has no directory under \\UNCPATHTOUSERPROFILES, any of the above solutions and the code in the question will throw an error when Get-ChildItem tries to enumerate that directory. You could fix that by checking that the directory exists first or passing -ErrorAction SilentlyContinue to Get-ChildItem, or you could enumerate \\UNCPATHTOUSERPROFILES and filter on those that occur in Users.csv...
$Shell = New-Object -ComObject WScript.Shell
try
{
# Performs faster filtering and eliminates duplicate user rows
$UsersTable = Get-Content C:\temp\Users.csv `
| Where-Object { $_ -notmatch '^\s*$' } `
| Group-Object -AsHashTable
# Get immediate child directories of \\UNCPATHTOUSERPROFILES
Get-ChildItem -Path \\UNCPATHTOUSERPROFILES -Directory -PipelineVariable 'UserDirectory' `
<# Filter for user directories specified in Users.csv #> `
| Where-Object { $UsersTable.ContainsKey($UserDirectory.Name) } `
<# Get *.lnk descendant files of the user directory #> `
| Get-ChildItem -Recurse -Include *.lnk -File `
| ForEach-Object -Process {
[PSCustomObject]#{
ShortcutName = $_.Name;
Target = $Shell.CreateShortcut($_).targetpath
User = $UserDirectory.Name
} `
} `
| Export-Csv c:\temp\report.csv -NoTypeInformation
}
finally
{
[Runtime.InteropServices.Marshal]::ReleaseComObject($Shell) | Out-Null
}

Exporting Object and strings to CSV using Powershell

The purpose of this code is to transfer files from one location to another and to log whether the transfer was a success or a failure.
Everything works except I am having issues with the log. I want the log to be in CSV format and there to be 3 columns: success/failure, from location, and to location. This is outputting the results all into rows with one column.
I've tried the Export-Csv option but that looks for objects/properties so only displays the length(I have strings too). Add-content works but there is only one column. Any suggestions?
#LOCATION OF CSV
$csv = Import-Csv C:\test2.csv
#SPECIFY DATE (EXAMPLE-DELETE FILES > 7 YEARS. 7 YEARS=2555 DAYS SO YOU WOULD ENTER "-2555" BELOW)
$Daysback = "-1"
#FILE DESTINATION
$storagedestination = "C:\Users\mark\Documents\Test2"
#LOG LOCATION
$loglocation = "C:\Users\mark\Documents\filetransferlog.csv"
$s = "SUCCESS"
$f = "FAIL"
$CurrentDate = Get-Date
foreach ($line in $csv) {
$Path = $line | Select-Object -ExpandProperty FullName
$DatetoDelete = $CurrentDate.AddDays($DaysBack)
$objects = Get-ChildItem $Path -Recurse | Select-Object FullName, CreationTime, LastWriteTime, LastAccessTime | Where-Object { $_.LastWriteTime -lt $DatetoDelete }
foreach ($object in $objects) {
try
{
$sourceRoot = $object | Select-Object -ExpandProperty FullName
Copy-Item -Path $sourceRoot -Recurse -Destination $storagedestination
Remove-Item -Path $sourceRoot -Force -Recurse
$temp = $s, $sourceRoot, $storagedestination
$temp | add-content $loglocation
}
catch
{
$temp2 = $f, $sourceRoot, $storagedestination
$temp2 | add-content $loglocation
}
}
}
All your | Select-Object -ExpandProperty are superfluous, simply attach the property name to the variable name => $Path = $line.FullName
Why calculate $DatetoDelete inside the foreach every time?
Output the success/fail to a [PSCustomObject] and gather them in a variable assigned directly to the foreach.
Untested:
$csv = Import-Csv C:\test2.csv
$Daysback = "-1"
$destination = "C:\Users\mark\Documents\Test2"
$loglocation = "C:\Users\mark\Documents\filetransferlog.csv"
$s = "SUCCESS"
$f = "FAIL"
$CurrentDate = Get-Date
$DatetoDelete = $CurrentDate.Date.AddDays($DaysBack)
$Log = foreach ($line in $csv) {
$objects = Get-ChildItem $line.FullName -Rec |
Where-Object LastWriteTime -lt $DatetoDelete
foreach ($object in $objects) {
$Result = $s
$sourceRoot = $object.FullName
try {
Copy-Item -Path $sourceRoot -Recurse -Destination $destination
Remove-Item -Path $sourceRoot -Recurse -Force
} catch {
$Result = $f
}
[PSCustomObject]#{
'Success/Fail' = $Result
Source = $sourceRoot
Destination = $destination
}
}
}
$Log | Export-Csv $loglocation -NoTypeInformation

Powershell get total size of files which user owns

I need a powershell script, which will go through all users in system and will find total size of all files which any user own... I have script which is going through all users, but then I've no idea to continue with counting total size which user owns for each user
Here is a script, which I`ve now:
$users = Get-WmiObject -class Win32_UserAccount
foreach($user in $users) {
$name = $user.Name
$fullName = $user.FullName;
if(Test-Path "C:\Users\$name") {
$path = "C:\Users\$name"
} else {
$path = "C:\Users\Public"
}
$dirSize = (Get-ChildItem $path -recurse | Measure-Object -property length -sum)
"{0:N2}" -f ($dirSize.sum / 1Gb) + " Gb"
echo "$dirSize"
Add-Content -path "pathototxt..." -value "$name $fullName $path"
}
I would be more than happy If somebody know the answer and tell me it...
Thank you
If there's a lot of files, you might want to consider:
$oSIDs = #{}
get-childitem <filespec> |
foreach {
$oSID = $_.GetAccessControl().Sddl -replace '^o:(.+?).:.+','$1'
$oSIDs[$oSID] += $_.length
}
Then resolve the SIDs when you're done. Parsing the owner SID or well-know security principal ID from the SDDL string saves the provider from having to do a lot of repetitive name resolution to give you back the "friendly" names.
I have no idea what you're asking for here.
"to continue with counting total size which user owns for each user". huh? Do want to check every file on the system or just the userfolder as you currently do?
Your script works fine if you just tweak it to include the filesize in the output. Personally I'd consider using a csv to store this because not all users will have e.g. a full name(admin, guest etc.). Also, atm. your script is counting the public folder multiple times(each time a user doesn't have a profile). E.g. admin(if it has never logged in), guest etc. might both get it specified.
Updated script that outputs both textfile and csv
$users = Get-WmiObject -class Win32_UserAccount
$out = #()
#If you want to append to a csv-file, replace the $out line above with the one below
#$out = Import-Csv "file.csv"
foreach($user in $users) {
$name = $user.Name
$fullName = $user.FullName;
if(Test-Path "C:\Users\$name") {
$path = "C:\Users\$name"
} else {
$path = "C:\Users\Public"
}
$dirSize = (Get-ChildItem $path -Recurse -ErrorAction SilentlyContinue | ? { !$_.PSIsContainer } | Measure-Object -Property Length -Sum)
$size = "{0:N2}" -f ($dirSize.Sum / 1Gb) + " Gb"
#Saving as textfile
#Add-Content -path "pathototxt..." -value "$name $fullName $path $size"
Add-Content -path "file.txt" -value "$name $fullName $path $size"
#CSV-way
$o = New-Object psobject -Property #{
Name = $name
FullName = $fullName
Path = $path
Size = $size
}
$out += $o
}
#Exporting to csv format
$out | Export-Csv "file.csv" -NoTypeInformation
EDIT: Another solution using the answer provided by #mjolinor and #C.B. modified to scan your c:\ drive while excluding some "rootfolders" like "program files", "windows" etc. It exports the result to a csv file ready for Excel.:
$oSIDs = #{}
$exclude = #("Program Files", "Program Files (x86)", "Windows", "Perflogs");
Get-ChildItem C:\ | ? { $exclude -notcontains $_.Name } | % { Get-ChildItem $_.FullName -Recurse -ErrorAction SilentlyContinue | ? { !$_.PSIsContainer } } | % {
$oSID = $_.GetAccessControl().Sddl -replace '^o:(.+?).:.+','$1'
$oSIDs[$oSID] += $_.Length
}
$out = #()
$oSIDs.GetEnumerator() | % {
$user = (New-Object System.Security.Principal.SecurityIdentifier($_.Key)).Translate([System.Security.Principal.NTAccount]).Value
$out += New-Object psobject -Property #{
User = if($user) { $user } else { $_.Key }
"Size(GB)" = $oSIDs[$_.Key]/1GB
}
}
$out | Export-Csv file.csv -NoTypeInformation -Delimiter ";"