Kerberos error in long running Exchange powershell skript after 10 hours - powershell

Question

I implemented a powershell script, which assigns Exchange settings to our user mailboxes (Exchange 2016). As we have a lot of mailboxes and assigning settings is slow, the script would run more then 15 hours. However after about 10 hours I get the following error:
Processing data for a remote command failed with the following error message: Error occurred during the Kerberos response.
[Server=XXXXX, TimeStamp = 74/2018 01:25:49]
For more information, see the about_Remote_Troubleshooting Help topic.
At C:\Users\ACCOUNT\AppData\Local\Temp\tmp_cj3akhk4.osq\tmp_cj3akhk4.osq.psm1:77943 char:9
+ $steppablePipeline.End()
+ ~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (XXXX:String) [], PSRemotingTransportException
+ FullyQualifiedErrorId : JobFailure
+ PSComputerName : XXXX
My script retires the operation and after two retries (which fail) an authentication prompt is shown. There I can enter the password of the service account and the script continues. However this dialog is only visible if I run the script in a PS command prompt. If the script is started as Windows Task, it just hangs and does not continue.
The connection to Exchange is opened and imported with the following code. The code can either connect to our on premises Exchange or Exchange online based on the passed parameter. The problem is currently only happening, when connected to our local (on premises) Exchange infrastructure.
Function Connect-Exchange{
PARAM(
[parameter(Mandatory=$false)]
[String]$TargetExchange = 'Local'
)
BEGIN{}
PROCESS{
if ($ExchangeSessionInfo.Session -and $ExchangeSessionInfo.Type -eq $TargetExchange -and $ExchangeSessionInfo.Session.State -eq 'Opened'){
# Nothing to do, we are already connected.
Write-Log "Exchange connection type $($TargetExchange) already established, nothing to do."
} else {
if ($ExchangeSessionInfo.Session -and $ExchangeSessionInfo.Type -ne $TargetExchange -and $ExchangeSessionInfo.Session.State -eq 'Opened'){
# We have a open session with the wrong type. We close it.
Remove-PSSession $ExchangeSessionInfo.Session
$ExchangeSessionInfo.Session = $null
$ExchangeSessionInfo.Status = 'undefined'
$ExchangeSessionInfo.Type = ''
}
# We close all other existing Exchange sessions we created.
get-pssession -Name "Exchange" -ErrorAction SilentlyContinue | remove-pssession
# Now connect to the requestes Exchange infrastructure and import session.
$Connected = $False
$RetryCount = 5
do{
try {
If ($TargetExchange -eq 'Local'){
$ExchangeServer = Get-Random -InputObject $LocalExchangeConfig.ExchangeServers
$ExchangeSessionInfo.Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "http://$($ExchangeServer)/PowerShell/" -Credential $EOCredentials -Authentication Kerberos -Name "Exchange"
} else {
$ExchangeSessionInfo.Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri 'https://ps.protection.outlook.com/powershell-liveid/' -Credential $EOCredentials -Authentication Basic -AllowRedirection -Name "Exchange"
}
$Res = Import-PSSession $ExchangeSessionInfo.Session -WarningAction SilentlyContinue -AllowClobber
# Store Exchange status in session variable.
$Connected = $True
$ExchangeSessionInfo.Status = 'connected'
$ExchangeSessionInfo.Type = $TargetExchange
} catch {
$err = Write-Error -err $error -msg "Could not connect to Exchange server type '$($TargetExchange)' (Retries left: $($RetryCount))." -Break $false
get-pssession -Name "Exchange" -ErrorAction SilentlyContinue | remove-pssession
$RetryCount -= 1
}
} while (!$Connected -and ($RetryCount -gt 0))
# If we do not have connection here, this is an error.
if (!$Connected) {
$ExchangeSessionInfo.Session = $null
$ExchangeSessionInfo.Status = 'undefined'
$ExchangeSessionInfo.Type = ''
throw "No connection to Exchange server (type: $($TargetExchange)) could be established."
} else {
# Get list of available mailbox DBs including mailbox count and create hashtable to store statistics. We only have to get it the first time.
if (($MailboxDBList.count -eq 0) -and ($TargetExchange -eq 'Local')){
Write-Log "Getting current Exchange DB configuration and mailbox count. Takes a moment."
$MailboxDBList = Get-MailboxDBCount -Type $LocalExchangeConfig.DistributeMailboxes
}
}
}
}
END{
return $ExchangeSessionInfo
}
}
The following code is applying a predefined set of Exchange settings:
...
$TryCount = 0
$Done = $false
do{
# It takes a while after enabling mailbox until settings can be applied. So we need to retry.
try{
# If we need to execute a setting several times.
if ($MailboxSetting.LoopOver){
# We have a loop value (array).
foreach ($LoopValue in $MailboxSetting.LoopOver){
# Copy parameter as we have to change a value (loop value).
$TempParams = $Params.PsObject.Copy()
#($Params.getenumerator()) |? {$_.Value -match '#LOOPVALUE#'} |% {$TempParams[$_.Key]=$LoopValue}
$res = & $MailboxSetting.Command -ErrorAction Stop #TempParams -WhatIf:$RunConfig.TestMode
}
} else {
# THE PROBLEM HAPPENS HERE
$res = & $MailboxSetting.Command -ErrorAction Stop #Params -WhatIf:$RunConfig.TestMode
}
# Write-Log "Setting command $($MailboxSetting.Command) executed successfully"
$Done = $true
} catch{
$tryCount++
$res = Write-Error -err $error -msg "Error applying mailbox settings, account: $($AccountDetails.sAMAccountName), retry count: $($TryCount)" -Break $false
Start-Sleep -s $(($Retires-$TryCount) * 5)
}
} while ((!$done) -and ($tryCount -lt $Retires))
...
I am sure the error is not related to the code, because the script runs for hours without a problem and applies all settings. However after a around 10 hours it seems the Kerberos ticket expires and then the script cannot longer access Exchange without a re-login.
Is there a way to keep the Kerberos ticket from expiring or renew it?
Any help would be appreciated.

Answer 1

I think you are hitting the domain security policy (group policy object - GPO) => security settings/account policy/Kerberos policy restriction.
The two valid options for you are:
Maximum lifetime for user ticket => the default value is 10 hours
Maximum lifetime for user ticket renewal => the default value is 7 days (this is the period within which the ticket can be renewed).
Is there a way to keep the Kerberos ticket from expiring or renew it?
For the first questions you "just" need to adjust the maximum lifetime for user ticket setting to value as you deem appropriate.
The second one is more tricky. I would just purge all kerberos tickets via the powershell. For more - viewing and purging cached kerberos tickets which would get you a new one.
If the ticket can be renewed you have to check the RENEABLE flag - you wan view it via kinit. Perhaps kinit -R could be enough for ticket renewal. (I did not do this my self) You could also renew it via kerberos for windows
Edit -- adding klist purge to purge all Kerberos tickets so it can be renewed.
As you have klist then you can purge all tickets via must be run in elevated powershell prompt
(all credits to JaredPoeppelman):
Get-WmiObject Win32_LogonSession | Where-Object {$_.AuthenticationPackage -ne 'NTLM'} | ForEach-Object {klist.exe purge -li ([Convert]::ToString($_.LogonId, 16))}
Then check if your TGT was updated via:
klist tgt
Note: you must use FQDN name everywhere!

Answer 2

Thanks for your suggestion. In a first try I will extend my code as follows and try to reestblisch a new Exchange connection. Needs 10 h runnig the script in order to see if this works.
I am not able to influence the domain security Policy, additionally as I do not know how long the script runs, it will be difficult to set a value.
On my Windows 2016 the command "kinit" ist not recognized. Possibly I need to install additional modules/roles.
...
$TryCount = 0
$Done = $false
do{
# It takes a while after enabling mailbox until settings can be applied. So we need to retry.
try{
# If we need to execute a setting several times.
if ($MailboxSetting.LoopOver){
# We have a loop value (array).
foreach ($LoopValue in $MailboxSetting.LoopOver){
# Copy parameter as we have to change a value (loop value).
$TempParams = $Params.PsObject.Copy()
#($Params.getenumerator()) |? {$_.Value -match '#LOOPVALUE#'} |% {$TempParams[$_.Key]=$LoopValue}
$res = & $MailboxSetting.Command -ErrorAction Stop #TempParams -WhatIf:$RunConfig.TestMode
}
} else {
$res = & $MailboxSetting.Command -ErrorAction Stop #Params -WhatIf:$RunConfig.TestMode
}
# Write-Log "Setting command $($MailboxSetting.Command) executed successfully"
$Done = $true
} catch{
$tryCount++
$res = Write-Error -err $error -msg "Error applying mailbox settings, account: $($AccountDetails.sAMAccountName), retry count: $($TryCount)" -Break $false
Start-Sleep -s $(($Retires-$TryCount) * 5)
try{
# We may have lost the Kerberos ticket, reconnect to Exchange.
$ConnectionType = $ExchangeSessionInfo.Type
Disconnect-Exchange
Connect-Exchange -TargetExchange $ConnectionType
} catch {}
}
} while ((!$done) -and ($tryCount -lt $Retires))
...