Split Powershell script into two separate parts - powershell

Question

I've got this script that connects to Sharepoint Online, indexes all the files and folders, downloads them all in a systematic fashion and churns out a .csv with the name of file, folders, size, path, etc.
For various reasons I've ended up in a situation where I've got all the data, but the metadata is corrupted (the .csv file aforementioned).
Unfortunately re running the whole script just for that isn't really an option, as that would require around 90 hours.
I've been trying to break the code down in order to remove the "download files" functions and just keep the part that generates the .csv, but so far without luck.
I've found the Function that seem to be in charge of it (WriteLog), but I'm struggling to separate it from the rest.
P.S. The code is not mine, I've inherited it from a developer I haven't got access to (unfortunately)
Please find the code below:
param(
[Parameter(Mandatory = $true)]
[string]$srcUrl,
[Parameter(Mandatory = $true)]
[string]$username,
[Parameter(Mandatory = $false,HelpMessage = "From Date: (dd/mm/yyyy)")]
[string]$fromDate,
[Parameter(Mandatory = $false,HelpMessage = "To Date: (dd/mm/yyyy)")]
[string]$toDate,
[Parameter(Mandatory = $true)]
[string]$folderPath,
[Parameter(Mandatory = $true)]
[string]$csvPath
) #end param
cls
#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\SharePoint Online Management Shell\Microsoft.Online.SharePoint.PowerShell\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\SharePoint Online Management Shell\Microsoft.Online.SharePoint.PowerShell\Microsoft.SharePoint.Client.Runtime.dll"
$global:OutFilePath = -join ($csvPath,"\Documents.csv")
$global:OutFilePathError = -join ($csvPath,"\ErrorLog_GetDocuments.csv")
$header = "Title,Type,Parent,Name,Path,FileSize(bytes),Created,Created by,Modified,Modified by,Matterspace title,Matterspace url"
$srcLibrary = "Documents"
$securePassword = Read-Host -Prompt "Enter your password: " -AsSecureString
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials ($username,$securePassword)
$sUrl = [System.Uri]$srcUrl
$domainUrl = -join ("https://",$sUrl.Host)
function WriteLog
{
param(
[Parameter(Mandatory = $true)] $title,$type,$folderName,$name,$path,$fileSize,$created,$createdby,$modifed,$modifiedby,$matterspacetitle,$materspaceUrl
)
$nowTime = Get-Date -Format "dd-MMM-yy,HH:mm:ss"
$folderName = $folderName.Replace(",","|") ### sometime folder / file name has comma so replace it with something
$name = $name.Replace(",","|")
#$path = $path.Replace(",","|")
$title=[System.String]::Concat("""""""$title""""""")
$type=[System.String]::Concat("""""""$type""""""")
$folderName=[System.String]::Concat("""""""$folderName""""""")
$name=[System.String]::Concat("""""""$name""""""")
$path=[System.String]::Concat("""""""$path""""""")
$fileSize=[System.String]::Concat("""""""$fileSize""""""")
$created=[System.String]::Concat("""""""$created""""""")
$createdby=[System.String]::Concat("""""""$createdby""""""")
$modified=[System.String]::Concat("""""""$modified""""""")
$modifiedby=[System.String]::Concat("""""""$modifiedby""""""")
$matterspacetitle=[System.String]::Concat("""""""$matterspacetitle""""""")
$materspaceUrl=[System.String]::Concat("""""""$materspaceUrl""""""")
$lineContent = "$("$title"),$($type),$($folderName),$($name),$($path),$($fileSize),$($created),$($createdby),$($modified),$($modifiedby),$($matterspacetitle),$($materspaceUrl)"
Add-Content -Path $global:OutFilePath -Value "$lineContent"
}
#Function to get all files of a folder
Function Get-FilesFromFolder([Microsoft.SharePoint.Client.Folder]$Folder,$SubWeb,$MTitle)
{
Write-host -f Yellow "Processing Folder:"$Folder.ServerRelativeUrl
$folderItem = $Folder.ListItemAllFields
#$srcContext.Load($f)
$Ctx.Load($folderItem)
$Ctx.ExecuteQuery()
#Get All Files of the Folder
$Ctx.load($Folder.files)
$Ctx.ExecuteQuery()
$authorEmail = $folderItem["Author"].Title
$editorEmail = $folderItem["Editor"].Title
$filepath = $folderItem["FileDirRef"]
if([string]::IsNullOrEmpty($filepath))
{
$filepath=$Folder.ServerRelativeUrl
}
$created = $folderItem["Created"]
$modified = $folderItem["Modified"]
$title = $folderItem["Title"]
if ([string]::IsNullOrEmpty($title))
{
$title = "Not Specified"
}
#$fileSize = $fItem["File_x0020_Size"]
$fileName = $Folder.Name
#list all files in Folder
write-host $Folder.Name
$splitString=$Folder.ServerRelativeUrl -split('/')
$dirUrl="";
write-host $splitString.Length
$parentUrl=""
For($i=3; $i -le $splitString.Length;$i++)
{
if($splitString[$i] -notcontains('.'))
{
Write-Host $i
Write-Host $splitString[$i]
$dirUrl=-join($dirUrl,"\",$splitString[$i])
$parentUrl=-join($parentUrl,"\",$splitString[$i+1])
}
}
$dirPath = -join ($folderPath,$dirUrl)
WriteLog $title "Folder" $parentUrl.TrimEnd('\') $fileName $filepath 0 $created $authorEmail $modified $editorEmail $MTitle $SubWeb
write-host $dirPath
if (-not (Test-Path -Path $dirPath))
{
New-Item -ItemType directory -Path $dirPath
}
ForEach ($File in $Folder.files)
{
try{
$remarkDetail = ""
$replacedUser = ""
$fItem = $File.ListItemAllFields
#$srcContext.Load($f)
$Ctx.Load($fItem)
$Ctx.ExecuteQuery()
$authorEmail = $fItem["Author"].Email
$editorEmail = $fItem["Editor"].Email
$filepath = $fItem["FileDirRef"]
$fileSizeBytes = $fItem["File_x0020_Size"];
$fileSize = ($fileSizeBytes) / 1MB
$fileName = $fItem["FileLeafRef"]
$title = $fItem["Title"]
$filecreated = $fitem["Created"]
$fileModified = $fitem["Modified"]
$FileUrl = $fItem["FileRef"]
$Fname=$File.Name
if ([string]::IsNullOrEmpty($title))
{
$title = "Not Specified"
}
#$title,$type, $folderName,$name,$path,$fileSize,$created,$createdby,$modifed,$modifiedby,$matterspacetitle,$materspaceUrl
$dateToCompare = Get-Date (Get-Date -Date $fileModified -Format 'dd/MM/yyyy')
#Get the File Name or do something
if (($dateToCompare -ge $startDate -and $dateToCompare -le $endDate) -or ($startDate -eq $null -and $endDate -eq $null))
{
$downloadUrl = -join ($dirPath,$File.Name)
$fromfile = -join ($domainUrl,$FileUrl)
Write-Host "Downloading the file from " $fromfile -ForegroundColor Cyan
try{
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials ($username,$securePassword)
$webclient.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED","f")
$webclient.DownloadFile($fromfile,$downloadUrl)
$webclient.Dispose()
}
catch{
$ErrorMessage=$_.Exception.Message
$ErrorMessage = $ErrorMessage -replace "`t|`n|`r",""
$ErrorMessage = $ErrorMessage -replace " ;|; ",";"
$lineContent = "$($Fname),$($fromfile ),$($ErrorMessage)"
Add-Content -Path $global:OutFilePathError -Value "$lineContent"
Write-Host "Skipping the file and recalling the function" -ForegroundColor Blue
}
WriteLog $title "File" $Folder.Name $fileName $FileUrl $fileSize $created $authorEmail $modified $editorEmail $MTitle $SubWeb
Write-host -f Magenta $File.Name
}
else
{
Write-Host "Skipping the matterspace :" $title " as the matterspace was not in the date range" -ForegroundColor Blue
}
}
catch{
$ErrorMessage=$_.Exception.Message
$ErrorMessage = $ErrorMessage -replace "`t|`n|`r",""
$ErrorMessage = $ErrorMessage -replace " ;|; ",";"
$lineContent = "$($Fname),$($fromfile ),$($ErrorMessage)"
Add-Content -Path $global:OutFilePathError -Value "$lineContent"
}
}
#Recursively Call the function to get files of all folders
$Ctx.load($Folder.Folders)
$Ctx.ExecuteQuery()
#Exclude "Forms" system folder and iterate through each folder
ForEach($SubFolder in $Folder.Folders | Where {$_.Name -ne "Forms"})
{
Get-FilesFromFolder -Folder $SubFolder -SubWeb $SubWeb -Mtitle $MTitle
}
}
Function Get-SPODocLibraryFiles()
{
param
(
[Parameter(Mandatory=$true)] [string] $SiteURL,
[Parameter(Mandatory=$true)] [string] $LibraryName
)
#Setup the context
$Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$Ctx.Credentials = $credentials
$srcWeb = $Ctx.Web
$childWebs = $srcWeb.Webs
$Ctx.Load($childWebs)
$Ctx.ExecuteQuery()
foreach ($childweb in $childWebs)
{
try
{
#Get the Library and Its Root Folder
$Library=$childweb.Lists.GetByTitle($LibraryName)
$Ctx.Load($Library)
$Ctx.Load($Library.RootFolder)
$Ctx.ExecuteQuery()
#Call the function to get Files of the Root Folder
if($childweb.Url.ToLower() -notlike "*ehcontactus*" -and $childweb.Url.ToLower() -notlike "*ehfaqapp*" -and $childweb.Url.ToLower() -notlike "*ehquicksearch*" -and $childweb.Url.ToLower() -notlike "*ehsiteapps*" -and $childweb.Url.ToLower() -notlike "*ehsitelist*" -and $childweb.Url.ToLower() -notlike "*ehwelcomeapp*" -and $childweb.Url.ToLower() -notlike "*ehimageviewer*")
{
Get-FilesFromFolder -Folder $Library.RootFolder -SubWeb $childweb.Url -MTitle $childweb.Title
}
}
catch{
write-host "Skipping the matterpsace as the library does not exists" -ForegroundColor Blue
}
}
}
#Config Parameters
#$SiteURL= "https://impigerspuat.sharepoint.com/sites/ELeave/Eleave1/adminuat#impigerspuat.onmicrosoft.com"
$LibraryName="Documents"
#$securePassword = Read-Host -Prompt "Enter your password: " -AsSecureString
#Call the function to Get All Files from a document library
if (-not ([string]::IsNullOrEmpty($fromDate)))
{
$startDate = Get-Date (Get-Date -Date $fromDate -Format 'dd/MM/yyyy')
}
else
{
$startDate = $null;
}
if (-not ([string]::IsNullOrEmpty($toDate)))
{
$endDate = Get-Date (Get-Date -Date $toDate -Format 'dd/MM/yyyy')
}
else
{
$endDate = $null
}
Get-SPODocLibraryFiles -SiteURL $srcUrl -LibraryName $LibraryName

Answer 1

Have you tried running just that function and giving it the parameters it's requesting in the function?
Copy the code into a WriteLog.ps1 file and then call the script file with the parameters.
ie.
Writelog.ps1 $srcUrl $username $fromDate $toDate $folderPath $csvPath
Obviously, inputting data in place of the variables.
FWIW, pulling relevant pieces of code out of someone else's scripts is a great skill to practice. Everything you want to do has been done before, but you might have to break down someone else's work before it fits your exact enviornment.

Answer 2

Unfortunately it looks like you have to do this the old fashion way. The problem is the author is outputting to the log (csv) as the files are being downloaded. As opposed to downloading to a staging area first...
I suggest setting an early break-point in the code then stepping through to see exactly how it's flowing. That should give you a general idea, and enough info to start writing refactored code.
Reverse engineering is always tough, be prepared it will be methodical exercise so say the least.

Answer 3

Bad news: this will be an iterative process, not a single 'solve'. Nothing "wrong" with that code, but there are a few design choices that make this a challenge. It's not indented consistently and it weaves through all the variable assignments in slightly different ways. Looks better than most of my code, I'm just telling you what makes it a challenge.
Good news: At least that WriteLog function is separate. And it's really just adding content to the .csv file defined in this variable assigned here:
$global:OutFilePath = -join ($csvPath,"\Documents.csv")
(line 20 in my copy)
*
RECOMMENDATION: (this is an approach, just a guide to your final solution)
Take that existing code and drop it in an IDE to help you visually. The Windows Powershell ISE is adequate, but I would highly recommend VSCode.
Comment out that last line:
Get-SPODocLibraryFiles -SiteURL $srcUrl -LibraryName $LibraryName
So you can retain any of the other context from the script you actually want to keep.
Create a separate function named something like:
function Get-FilesFromLocalFolder ($localdir, $SubWeb, $MTitle)
to use instead of the existing function Get-FilesFromFolder. That way you can iterate through whatever directories you need, get the files, and assign variables to pass as parameters. Then when you call WriteLog, it will look very similar. Those last two parameters ($SubWeb, $MTitle) are passed just because WriteLog needs them. You could make them your own labels, or you could remove them and make them optional in WriteLog.
You could start by hard-coding values in each of required parameters for the function, and then run it to see if the output is working.
It will take you some iterations (agree with #Steven) and it is definitely a valuable exercise (agree with #TheIdesOfMark). :)