How do I use a switch statement with while loop - powershell

Question

I am trying to add a user to the Remote Desktop group using Powershell.
It seems it's not breaking the loop and doesn't execute further statements.
Can you please help me where the issue is:
$remote = ""
While($remote -ne "Y" ){
$remote = read-host "Do you need to add anyone to the Remote Desktop group (y/n)?"
Switch ($remote)
{
Y {
$remoteuser = ""
while ( ($remoteuser -eq "") -or ($UserExists -eq $false) )
{
$remoteuser = Read-Host "Enter the username that needs to be in the group"
Write-Host "User inputted $remoteuser"
sleep -Seconds 2
try {
Get-ADUser -Identity $remoteuser -Server <server-FQDN>
$UserExists = $true
Write-Host "$remoteuser found!"
sleep 5
}
catch [Microsoft.ActiveDirectory.Management.ADIdentityResolutionException] {
Write-Host "User does not exist."
$UserExists = $false
}
catch {
Write-Host "Username is blank"
$UserExists = $false
}
}
}
N {Write-Host "No user accounts will be added to the Remote Desktop Users group. Restart your PC."}
default {Write-Host "Only Y/N are Valid responses"}
}
}
<further statements>