Azure Service Fabric - Automate Deletion/Re-creation of a service - azure-service-fabric

Question

We have a service in our Azure Service Fabric Cluster which seems to be freezing and losing connection with our database about once every 48 hours. Until I get a developer to look into the issue, my workaround has been to go and Delete the service via the Service Fabric Explorer and then immediately re-create it. This fixed the issue temporarily until it freezes up again.
My question is whether there is anyway I can automate this process? It will be at least a month or two before I can get a developer to look into it so I'm looking for away to run the process automatically once a day.

Answer 1

Unfortunately there is no scheduling mechanism in Service Fabric to do this kind of operation.
A solution for you is running a script that connects to SF and execute the Restart Code Package either via powershell or via API.
For powershell you can use an Azure Automation Runbook or use a Azure Functions to call the API in a schedule.
I think the powershell is easier, but both should work.
Restart-ServiceFabricDeployedCodePackage, as the name suggest, will force a shutdown and restart a process and all replicas hosted within it. No need to delete and recreate, you might miss configure the service.
The documentation shows a combination of parameters that can be used together, in some cases, some parameters is required when used with others, the docs should highlight the matches, the result will be something like this:
Restart-ServiceFabricDeployedCodePackage -ApplicationName "fabric:/appname" -ServiceName "fabric:/appname/servicename" -PartitionId "b098c9f0-009a-458d-8b2d-8089fedcd014"
or a specific replica like this:
Restart-ServiceFabricDeployedCodePackage -ApplicationName "fabric:/repairs" -ServiceName "fabric:/repairs/web" -PartitionId "b098c9f0-009a-458d-8b2d-8089fedcd014" -ReplicaOrInstanceId 131896982398426643
.
The Restart-ServiceFabricPartition is also useful, have the same effects:
Restart-ServiceFabricPartition -RestartPartitionMode AllReplicasOrInstances -ServiceName "fabric:/appname/service" -PartitionId "b098c9f0-009a-458d-8b2d-8089fedcd014"
Restart-ServiceFabricPartition has become obsolete to move people to use Start-ServiceFabricPartitionRestart that is recommended for shutdown services when reliability is required, for example stateful services, it will avoid put down all replicas at same time.
Start-ServiceFabricPartitionRestart I haven't used it myself, but is what the recommendation suggests for stateful services.
The parameters combination is a bit tricky, I recommend you try with different combinations. In some cases it succeed but shows an error, not sure why!

Answer 2

Here's my script that I use to restart stateless services. It uses the Restart-ServiceFabricDeployedCodePackage mentioned above.
I've called it Restart-ServiceFabricServiceCodePackages.ps1 and you can just call it with a -ServiceName fabric:/application/service and adjust the wait time according to your service's normal start-up time.
Param (
[Parameter(Mandatory=$true)]
[uri] $ServiceName,
[int] $WaitBetweenNodesSeconds = 30
)
try {
Test-ServiceFabricClusterConnection | Out-Null
}
catch {
throw "Active connection to Service Fabric cluster required"
}
$serviceDescription = Get-ServiceFabricServiceDescription -ServiceName $ServiceName -ErrorAction SilentlyContinue
if (!$serviceDescription) {
throw "Invalid Service Fabric service name"
}
if ($serviceDescription.ServiceKind -ne "Stateless") {
throw "Unknown outcomes could occur for non-stateless services"
}
$applicationName = $serviceDescription.ApplicationName
$serviceTypeName = $serviceDescription.ServiceTypeName
$service = Get-ServiceFabricService -ServiceName $ServiceName -ApplicationName $applicationName
$application = Get-ServiceFabricApplication -ApplicationName $applicationName
$serviceType = Get-ServiceFabricServiceType -ServiceTypeName $serviceTypeName -ApplicationTypeName $application.ApplicationTypeName -ApplicationTypeVersion $application.ApplicationTypeVersion
$serviceManifestName = $serviceType.ServiceManifestName
$nodes = Get-ServiceFabricNode -StatusFilter Up
$nodes | Where-Object {
$nodeName = $_.NodeName
$hasService = $null
$hasApplication = Get-ServiceFabricDeployedApplication -NodeName $nodeName -ApplicationName $applicationName
if ($hasApplication) {
$hasService = Get-ServiceFabricDeployedServicePackage -NodeName $nodeName -ApplicationName $applicationName -ServiceManifestName $serviceManifestName -ErrorAction SilentlyContinue
}
return $hasApplication -and $hasService
} | ForEach-Object {
$nodeName = $_.NodeName
$codePackages = Get-ServiceFabricDeployedCodePackage -NodeName $nodeName -ApplicationName $applicationName -ServiceManifestName $serviceManifestName
$codePackages | ForEach-Object {
$codePackageName = $_.CodePackageName
$servicePackageActivationId = $_.ServicePackageActivationId
$codePackageInstanceId = $_.EntryPoint.CodePackageInstanceId
Write-Host "Restarting deployed package on $nodeName named $codePackageName (for service package id: $servicePackageActivationId and code package id: $codePackageInstanceId)"
$success = Restart-ServiceFabricDeployedCodePackage -NodeName $nodeName -ApplicationName $applicationName -ServiceManifestName $serviceManifestName -CodePackageName $codePackageName -CodePackageInstanceId $codePackageInstanceId -ServicePackageActivationId $servicePackageActivationId -CommandCompletionMode Invalid
if ($success) {
Write-Host "Successfully restored deployed package on $nodeName" -ForegroundColor Green
}
Write-Host "Waiting for $WaitBetweenNodesSeconds seconds for previous node to restart before continuing"
Start-Sleep -Seconds $WaitBetweenNodesSeconds
$retries = 0
$service = Get-ServiceFabricService -ServiceName $ServiceName -ApplicationName $applicationName
while ($retries -lt 3 -and ($service.HealthState -ne "Ok" -or $service.ServiceStatus -ne "Active")) {
$service = Get-ServiceFabricService -ServiceName $ServiceName -ApplicationName $applicationName
$retries = $retries + 1
Write-Host "Waiting for an additional 15 seconds for previous node to restart before continuing because service state is not healthy"
Start-Sleep -Seconds 15
}
}
}