Azure Service Fabric IPv6 networking issues - azure-service-fabric

Question

We are having issues deploying our Service Fabric cluster to Azure and have it handle both IPv4
and IPv6 traffic.
We are developing an application that have mobile clients on iOS and Android which communicate with
our Service Fabric cluster. The communication consist of both HTTP traffic as well as TCP Socket communication.
We need to support IPv6 in order to have Apple accept the app in their App Store.
We are using ARM template for deploying to Azure as it seems the portal does not support configuring
load balancer with IPv6 configuration for Virtual Machine Scale Sets (ref: url). The linked page also states other limitations
to the IPv6 support, such as private IPv6 addresses cannot be deployed to VM Scale Sets. However according
to this page the possibility to assign private IPv6 to VM Scale Sets is available in preview
(although this was last updated 07/14/2017).
For this question I have tried to keep this as general as possible, and based the ARM template on a template found
in this tutorial. The template is called "template_original.json" and can be downloaded from
here. This is a basic template for a service fabric cluster with no security for simplicity.
I will be linking the entire modified ARM template in the bottom of this post, but will highlight the
main modified parts first.
Public IPv4 and IPv6 addresses that are associated with the load balancer. These are associated with their respective backend pools:
"frontendIPConfigurations": [
{
"name": "LoadBalancerIPv4Config",
"properties": {
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('lbIPv4Name'),'-','0'))]"
}
}
},
{
"name": "LoadBalancerIPv6Config",
"properties": {
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(parameters('lbIPv6Name'),'-','0'))]"
}
}
}
],
"backendAddressPools": [
{
"name": "LoadBalancerIPv4BEAddressPool",
"properties": {}
},
{
"name": "LoadBalancerIPv6BEAddressPool",
"properties": {}
}
],
Load balancing rules for frontend ports on respective public IP addresses, both IPv4 and IPv6.
This amounts to four rules in total, two per front end port. I have added port 80 for HTTP here and port 5607 for Socket connection.
Note that I have updated the backend port for IPv6 port 80 to be 8081 and IPv6 port 8507 to be 8517.
{
"name": "AppPortLBRule1Ipv4",
"properties": {
"backendAddressPool": {
"id": "[variables('lbIPv4PoolID0')]"
},
"backendPort": "[parameters('loadBalancedAppPort1')]",
"enableFloatingIP": "false",
"frontendIPConfiguration": {
"id": "[variables('lbIPv4Config0')]"
},
"frontendPort": "[parameters('loadBalancedAppPort1')]",
"idleTimeoutInMinutes": "5",
"probe": {
"id": "[concat(variables('lbID0'),'/probes/AppPortProbe1')]"
},
"protocol": "tcp"
}
},
{
"name": "AppPortLBRule1Ipv6",
"properties": {
"backendAddressPool": {
"id": "[variables('lbIPv6PoolID0')]"
},
/*"backendPort": "[parameters('loadBalancedAppPort1')]",*/
"backendPort": 8081,
"enableFloatingIP": "false",
"frontendIPConfiguration": {
"id": "[variables('lbIPv6Config0')]"
},
"frontendPort": "[parameters('loadBalancedAppPort1')]",
/*"idleTimeoutInMinutes": "5",*/
"probe": {
"id": "[concat(variables('lbID0'),'/probes/AppPortProbe1')]"
},
"protocol": "tcp"
}
},
{
"name": "AppPortLBRule2Ipv4",
"properties": {
"backendAddressPool": {
"id": "[variables('lbIPv4PoolID0')]"
},
"backendPort": "[parameters('loadBalancedAppPort2')]",
"enableFloatingIP": "false",
"frontendIPConfiguration": {
"id": "[variables('lbIPv4Config0')]"
},
"frontendPort": "[parameters('loadBalancedAppPort2')]",
"idleTimeoutInMinutes": "5",
"probe": {
"id": "[concat(variables('lbID0'),'/probes/AppPortProbe2')]"
},
"protocol": "tcp"
}
},
{
"name": "AppPortLBRule2Ipv6",
"properties": {
"backendAddressPool": {
"id": "[variables('lbIPv6PoolID0')]"
},
"backendPort": 8517,
"enableFloatingIP": "false",
"frontendIPConfiguration": {
"id": "[variables('lbIPv6Config0')]"
},
"frontendPort": "[parameters('loadBalancedAppPort2')]",
/*"idleTimeoutInMinutes": "5",*/
"probe": {
"id": "[concat(variables('lbID0'),'/probes/AppPortProbe2')]"
},
"protocol": "tcp"
}
}
Also added one probe per load balancing rule, but omitted here for clarity.
The apiVerison for VM Scale set is set to "2017-03-30" per recommendation from aforementioned preview solution.
The network interface configurations are configured according to recommendations as well.
"networkInterfaceConfigurations": [
{
"name": "[concat(parameters('nicName'), '-0')]",
"properties": {
"ipConfigurations": [
{
"name": "[concat(parameters('nicName'),'-IPv4Config-',0)]",
"properties": {
"privateIPAddressVersion": "IPv4",
"loadBalancerBackendAddressPools": [
{
"id": "[variables('lbIPv4PoolID0')]"
}
],
"loadBalancerInboundNatPools": [
{
"id": "[variables('lbNatPoolID0')]"
}
],
"subnet": {
"id": "[variables('subnet0Ref')]"
}
}
},
{
"name": "[concat(parameters('nicName'),'-IPv6Config-',0)]",
"properties": {
"privateIPAddressVersion": "IPv6",
"loadBalancerBackendAddressPools": [
{
"id": "[variables('lbIPv6PoolID0')]"
}
]
}
}
],
"primary": true
}
}
]
With this template I am able to successfully deploy it to Azure. Communication using IPv4 with the
cluster works as expected, however I am unable to get any IPv6 traffic through at all. This is the
same for both ports 80 (HTTP) and 5607 (socket).
When viewing the list of backend pools for the load balancer in the Azure portal it displays the
following information message which I have been unable to find any information about. I am unsure
if this affects anything in any way?
Backend pool 'loadbalanceripv6beaddresspool' was removed from Virtual machine scale set 'Node1'. Upgrade all the instances of 'Node1' for this change to apply Node1
load balancer error message
I am not sure why I cannot get the traffic through on IPv6. It might be that there is something I
have missed in the template, or some other error on my part? If any additional information is required
dont hesitate to ask.
Here is the entire ARM template. Due to the length and post length limitations I have not embedded it, but here is a Pastebin link to the full ARM Template (Updated).
Update
Some information regarding debugging the IPv6 connectivity. I have tried slightly altering the ARM template to forward the IPv6 traffic on port 80 to backend port 8081 instead. So IPv4 is 80=>80 and IPv6 80=>8081. The ARM template has been updated (see link in previous section).
On port 80 I am running Kestrel as a stateless web server. I have the following entries in the ServiceManifest.xml:
<Endpoint Protocol="http" Name="ServiceEndpoint1" Type="Input" Port="80" />
<Endpoint Protocol="http" Name="ServiceEndpoint3" Type="Input" Port="8081" />
I have been a bit unsure specifically which addresses to listen for in Kestrel. Using FabricRuntime.GetNodeContext().IPAddressOrFQDN always returns the IPv4 address. This is currently how we start it. For debugging this I currently get ahold of all the IPv6 addresses, and hardcoded hack for port 8081 we use that address. Fort port 80 use IPAddress.IPv6Any, however this always defaults to the IPv4 address returned by FabricRuntime.GetNodeContext().IPAddressOrFQDN.
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
var endpoints = Context.CodePackageActivationContext.GetEndpoints()
.Where(endpoint => endpoint.Protocol == EndpointProtocol.Http ||
endpoint.Protocol == EndpointProtocol.Https);
var strHostName = Dns.GetHostName();
var ipHostEntry = Dns.GetHostEntry(strHostName);
var ipv6Addresses = new List<IPAddress>();
ipv6Addresses.AddRange(ipHostEntry.AddressList.Where(
ipAddress => ipAddress.AddressFamily == AddressFamily.InterNetworkV6));
var listeners = new List<ServiceInstanceListener>();
foreach (var endpoint in endpoints)
{
var instanceListener = new ServiceInstanceListener(serviceContext =>
new KestrelCommunicationListener(
serviceContext,
(url, listener) => new WebHostBuilder().
UseKestrel(options =>
{
if (endpoint.Port == 8081 && ipv6Addresses.Count > 0)
{
// change idx to test different IPv6 addresses found
options.Listen(ipv6Addresses[0], endpoint.Port);
}
else
{
// always defaults to ipv4 address
options.Listen(IPAddress.IPv6Any, endpoint.Port);
}
}).
ConfigureServices(
services => services
.AddSingleton<StatelessServiceContext>(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseStartup<Startup>()
.UseUrls(url)
.Build()), endpoint.Name);
listeners.Add(instanceListener);
}
return listeners;
}
Here is the endpoints shown in the Service Fabric Explorer for one of the nodes: Endpoint addresses
Regarding the socket listener I have also altered so that IPv6 is forwarded to backend port 8517 instead of 8507. Similarily as with Kestrel web server the socket listener will open two listening instances on respective addresses with appropriate port.
I hope this information is of any help.

Answer 1

It turns out I made a very stupid mistake that is completely my fault, I forgot to actually verify that my ISP fully supports IPv6. Turns out they don't!
Testing from a provider with full IPv6 support works as it should and I am able to get full connectivity to the nodes in the Service Fabric cluster.
Here is the working ARM template for anyone that needs a fully working example of Service Fabric cluster with IPv4 and IPv6 support:
Not allowed to post pastebin links without a accompanied code snippet...
Update:
Due to length constraints the template could not be pasted in this thread in its entirety, however over on the GitHub Issues page for Service Fabric I crossposted this. The ARM template is posted as a comment in that thread, it will hopefully be available longer than the pastebin link. View it here.