What is the proper way to resolve a reference to a service fabric stateless service? - service

Question

I have been developing a system which is heavily using stateless service fabric services. I thought I had a good idea how things worked however I am doing something slightly different now and it is obvious my understanding is limited.
I have a cluster of 5 nodes and all my services have an instance count of -1 for simplicity currently. With everything on every node it means I can watch a single node for basic behavior correctness.
I just added a new service which needs an instance count of 1. However it seems I am unable to resolve this service correctly. Instead SF tries to resolve the service on each machine which fails for all machines except the one where the single service exists.
It was my assumption that SF would automagically resolve a reference to a service anywhere in the cluster. If that reference fails then it would automagically resolve a new reference and so on. It appears that this is not correct at least for the way I am currently doing things.
I can find an instance using code similar to this however what happens if that instance fails. How do I get another reference?
I can resolve for every call like this however that seems like a terrible idea when I really only want to resolve a IXyzService and pass that along.
This is how I am resolving services since I am using the V2 custom serialization.
var _proxyFactory = new ServiceProxyFactory(c =>
{
return new FabricTransportServiceRemotingClientFactory(
serializationProvider: new CustomRemotingSerializationProvider(Logger)
);
});
var location = new Uri("fabric:/xyz/abcService");
var proxy = _proxyFactory.CreateServiceProxy<TService>(location);
This does actually work however it appears to only resolve a service on the same machine. So ServiceA would resolve a reference to ServiceB on the same machine. However if ServiceB doesn't exist on the machine for a valid reason then the resolution would fail.
Summary:
What is the correct way for ServiceA to use the V2 custom serialization ServiceProxyFactory to resolve an interface reference to ServiceB wherever ServiceA and ServiceB are in the cluster?
Update:
The evidence it doesn't work is the call to resolve hangs forever. According to this link that is correct behavior because the service will eventually come up. However only 1 node ever resolved it correctly and that is the node where the single instance is active. I have tried several things even waiting 30 seconds just to make sure it wasn't an init issue.
var proxy = _proxyFactory.CreateServiceProxy<TService>(location);
// Never gets here except on the one node.
SomethingElse(proxy);
Listener code
This essentially follows the V2 custom serialization tutorial almost exactly.
var listeners = new[]
{
new ServiceInstanceListener((c) =>
{
return new FabricTransportServiceRemotingListener(c, this, null,
new CustomRemotingSerializationProvider(Logger));
})
};
public class HyperspaceRemotingSerializationProvider : IServiceRemotingMessageSerializationProvider
{
#region Private Variables
private readonly ILogger _logger;
private readonly Action<RequestInfo> _requestAction;
private readonly Action<RequestInfo> _responseAction;
#endregion Private Variables
public CustomRemotingSerializationProvider(ILogger logger, Action<RequestInfo> requestAction = null, Action<RequestInfo> responseAction = null)
{
_logger = logger;
_requestAction = requestAction;
_responseAction = responseAction;
}
public IServiceRemotingRequestMessageBodySerializer CreateRequestMessageSerializer(Type serviceInterfaceType, IEnumerable<Type> requestWrappedTypes,
IEnumerable<Type> requestBodyTypes = null)
{
return new RequestMessageBodySerializer(_requestAction);
}
public IServiceRemotingResponseMessageBodySerializer CreateResponseMessageSerializer(Type serviceInterfaceType, IEnumerable<Type> responseWrappedTypes,
IEnumerable<Type> responseBodyTypes = null)
{
return new ResponseMessageBodySerializer(_responseAction);
}
public IServiceRemotingMessageBodyFactory CreateMessageBodyFactory()
{
return new MessageBodyFactory();
}
}
Connection code
_proxyFactory = new ServiceProxyFactory(c =>
{
return new FabricTransportServiceRemotingClientFactory(
serializationProvider: new CustomRemotingSerializationProvider(Logger)
);
});
// Hangs here - tried different partition keys or not specifying one.
var proxy = _proxyFactory.CreateServiceProxy<TService>(location, ServicePartitionKey.Singleton);