Creating Secured standalone service fabric cluster using X509 Certificate - azure-service-fabric

Question

I am trying to create three Node secure Service fabric cluster using service fabric runtime 6.3.162.9494 with X.509 certificate.
The three machine are
Windows Server 2016 Standard version 1607 with GUI(build 14393.447)
Windows 10 Enterprise 2016 LTSB version 1607 (build 14393.447)
Windows 10 Enterprise 2016 LTSB version 1607 (build 14393.447)
The Service fabric package i have downloaded from http://go.microsoft.com/fwlink/?LinkId=730690
and i used ClusterConfig.X509.MultiMachine.json configuartion file.
I referred this (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-windows-cluster-x509-security) document to create the cluster.
First i have created a self signed certificate and installed on all machines in
Local Computer - Personal
Local Computer - Trusted Root Certification Authorities
Current User - Personal
The Properties i have configured in configuration file is like this:
{
"name": "SampleCluster",
"clusterConfigurationVersion": "1.0.0",
"apiVersion": "10-2017",
"nodes": [
{
"nodeName": "Server",
"iPAddress": "10.140.133.171",
"nodeTypeRef": "NodeType0",
"faultDomain": "fd:/dc1/r0",
"upgradeDomain": "UD0"
},
{
"nodeName": "Client1",
"iPAddress": "10.140.133.199",
"nodeTypeRef": "NodeType0",
"faultDomain": "fd:/dc1/r1",
"upgradeDomain": "UD1"
},
{
"nodeName": "Client2",
"iPAddress": "10.140.133.177",
"nodeTypeRef": "NodeType0",
"faultDomain": "fd:/dc1/r2",
"upgradeDomain": "UD2"
}
],
"properties": {
"diagnosticsStore":
{
"metadata": "Please replace the diagnostics file share with an actual file share accessible from all cluster machines. For example, \\\\machine1\\DiagnosticsStore.",
"dataDeletionAgeInDays": "21",
"storeType": "FileShare",
"connectionstring": "\\\\10.140.133.171\\DiagnosticsStore"
},
"security": {
"metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates. The thumbprint format is - d5 ec 42 3b 79 cb e5 07 fd 83 59 3c 56 b9 d5 31 24 25 42 64.",
"ClusterCredentialType": "X509",
"ServerCredentialType": "X509",
"CertificateInformation": {
"ClusterCertificateCommonNames": {
"CommonNames": [
{
"CertificateCommonName": "WindowsCertificate"
}
],
"X509StoreName": "My"
},
"ClusterCertificateIssuerStores": [
{
"IssuerCommonName": "WindowsCertificate",
"X509StoreNames" : "Root"
}
],
"ServerCertificateCommonNames": {
"CommonNames": [
{
"CertificateCommonName": "WindowsCertificate"
}
],
"X509StoreName": "My"
},
"ServerCertificateIssuerStores": [
{
"IssuerCommonName": "WindowsCertificate",
"X509StoreNames" : "Root"
}
],
"ReverseProxyCertificateCommonNames": {
"CommonNames": [
{
"CertificateCommonName": "WindowsCertificate"
}
],
"X509StoreName": "My"
},
"ClientCertificateThumbprints": [
{
"CertificateThumbprint": "86 4d 24 ea 57 29 08 a0 4d f4 9c 6d ec f6 04 5f 82 65 01 db",
"IsAdmin": false
},
{
"CertificateThumbprint": "86 4d 24 ea 57 29 08 a0 4d f4 9c 6d ec f6 04 5f 82 65 01 db",
"IsAdmin": true
}
],
"ClientCertificateCommonNames": [
{
"CertificateCommonName": "WindowsCertificate",
"IsAdmin": true
}
],
"ClientCertificateIssuerStores": [
{
"IssuerCommonName": "WindowsCertificate",
"X509StoreNames": "Root"
}
]
}
},
"nodeTypes": [
{
"name": "NodeType0",
"clientConnectionEndpointPort": "19000",
"clusterConnectionEndpointPort": "19001",
"leaseDriverEndpointPort": "19002",
"serviceConnectionEndpointPort": "19003",
"httpGatewayEndpointPort": "19080",
"reverseProxyEndpointPort": "19081",
"applicationPorts": {
"startPort": "20001",
"endPort": "20031"
},
"isPrimary": true
}
],
"fabricSettings": [
{
"name": "Setup",
"parameters": [
{
"name": "FabricDataRoot",
"value": "C:\\ProgramData\\SF"
},
{
"name": "FabricLogRoot",
"value": "C:\\ProgramData\\SF\\Log"
}
]
}
]
}
}
While creating the cluster i am getting this error the error i am getting

Answer 1

There can be many reasons for this. Even though powershell error says that certificate cannot be found, the real issue might be slightly different.
The most common problem can be that machine from which you are creating the cluster does not have connection to the node machines. I would start with verifying that.
Also, Run creation script one more time and check
C:\Users\Administrator\Desktop\Microsoft.Azure.ServiceFabric.WindowsServer.6.3.162.9494\DeploymentTraces
You will find more accurate error description there.
If the error is the same in the trace file, check access rights for the server certificate on each node. By default Service Fabric is running on NETWORK SERVICE user. You should allow access rights for this user.

Answer 2

The reason it is failing is quite clear from the error message.
Basically - you have to ensure that all your server/cluster certificates:
Are installed on EACH node (BEFORE you create the cluster)
Are accessible by "WINDOWS SERVICE" account (given you're using X509) - see MSDOC on how to do this