Installing Secure Standalone Service Fabric Dev Cluster - azure-service-fabric

Question

I'm trying to follow this:
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-windows-cluster-x509-security
I have edited ClusterConfig.X509.DevCluster.json and replaced localhost for my machine's IP. I have include my certificates in it and the security node is as follows:
"security": {
"metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates. The thumbprint format is - d5 ec 42 3b 79 cb e5 07 fd 83 59 3c 56 b9 d5 31 24 25 42 64.",
"ClusterCredentialType": "X509",
"ServerCredentialType": "X509",
"CertificateInformation": {
"ClusterCertificate": {
"Thumbprint": "xx xx xx xx dc c9 a1 2e ae 2d 68 90 8e 7d f0 1e 79 05 d6 6b",
"X509StoreName": "My"
},
"ServerCertificate": {
"Thumbprint": "xx xx xx xx dc c9 a1 2e ae 2d 68 90 8e 7d f0 1e 79 05 d6 6b",
"X509StoreName": "My"
},
"ReverseProxyCertificate": {
"Thumbprint": "xx xx xx xx ee 08 00 ea f0 69 7f 4f 2c 61 49 0c 28 20 11 8b",
"X509StoreName": "My"
}
}
},
My config appears valid:
ClusterConfigFilePath: ClusterConfig.json
DeploymentComponents extracted.
Trace folder doesn't exist. Creating trace folder: C:\SF-Install\DeploymentTraces
Running Best Practices Analyzer...
Best Practices Analyzer completed successfully.
LocalAdminPrivilege : True
IsJsonValid : True
IsCabValid :
RequiredPortsOpen : True
RemoteRegistryAvailable : True
FirewallAvailable : True
RpcCheckPassed : True
NoConflictingInstallations : True
FabricInstallable : True
DataDrivesAvailable : True
Passed : True
The installation times out with the following error:
Timed out waiting for Installer Service to complete for machine 192.168.168.114. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
CreateCluster Error: System.AggregateException: One or more errors occurred. ---> System.ServiceProcess.TimeoutException : Timed out waiting for Installer Service to complete for machine 192.168.168.114. Investigation order: FabricInstallerS
ervice -> FabricSetup -> FabricDeployer -> Fabric
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController installerSvc)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.RunFabricServices(List`1 machines, FabricPackageType fabricPackageType)
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.<CreateClusterAsyncInternal>d__7.MoveNext()
---> (Inner Exception #0) System.ServiceProcess.TimeoutException: Timed out waiting for Installer Service to complete for machine 192.168.168.114. Investigation order: FabricInstallerService -> FabricSetup -> FabricDeployer -> Fabric
at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(String machineName, ServiceController installerSvc)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )<---
I can install ClusterConfig.Unsecure.DevCluster.json perfectly fine.
I am on a fresh installation of Windows Server 2016.
My certificates were created with New-SelfSignedCertificate -DnsName "xxxx"
In the event log I'm getting
SecurityCredentials
AcquireCredentialsHandle(Microsoft Unified Security Protocol Provider) failed: 0x8009030d
From user NETWORK SERVICE

Answer 1

The problem was NETWORK SERVICE needs adding to ACL for certificates used by SF.
See "Install the certificates" here

Answer 2

For me, the Microsoft PS script to set the cert ACLs didn't work as I was using CNG certs which means $cert.PrivateKey returns null.
The solution for me was to use
certutil -store my certificate_thumbprint
to get the unique container name and then grant NETWORK SERVICE full control via GUI through the root share:
\\headlesshost\c$\programdata\microsoft\Crypto\Keys\unique_container_name