Slow performance in reliable collections with relatively large dataset - azure-service-fabric

Question

I have a reliable service with 3 partitions. The data is partitioned reasonably equally between the 3 - it's almost exact. I have just under 3.95 million entries of the following contract across the 3 partitions
[DataContract]
public class Item
{
[DataMember]
public string StandardisedText { get; set; }
[DataMember]
public string Vendor{ get; set; }
[DataMember]
public bool Card{ get; set; }
[DataMember]
public string Suburb { get; set; }
[DataMember]
public string State { get; set; }
[DataMember]
public string PostCode { get; set; }
[DataMember]
public float Latitude { get; set; }
[DataMember]
public float Longitude { get; set; }
[DataMember]
public string Level1 { get; set; }
[DataMember]
public string Level2 { get; set; }
[DataMember]
public string Level3 { get; set; }
[DataMember]
public string Level4 { get; set; }
}
I have an upstream aggregate service which reports back a count across all partitions with the following code (please ignore the terribleness of this code, its a quick and dirty POC to see how viable reliable collections are for reasonably large data sets).
[HttpGet("")]
public IActionResult GetAll()
{
try
{
_logger.LogError("Getting all count");
ServiceEventSource.Current.Message("Getting all count");
var settings = new FabricTransportRemotingSettings { OperationTimeout = TimeSpan.FromMinutes(30) };
var factory =
new ServiceProxyFactory(h =>
new FabricTransportServiceRemotingClientFactory(settings));
int total = 0;
foreach (var servicePartitionKey in PartitionKeys)
{
var proxy = factory.CreateServiceProxy<ITermTextService>(
new Uri("fabric:/Acme.Fabric.Refinery/Acme.Fabric.Refinery.RefineryStore"),
servicePartitionKey);
var count = proxy.Count().Result;
ServiceEventSource.Current.Message($"Searched partition {servicePartitionKey.Value} and found {count} values" );
_logger.LogInformation($"Searched partition {servicePartitionKey.Value} and found {count} values");
total += count;
}
return Ok(total);
}
catch (Exception excep)
{
_logger.LogError($"Error in get all {excep.Message} {excep.StackTrace}");
ServiceEventSource.Current.Message($"Error {excep.Message} {excep.StackTrace}");
}
return null;
}
The count code is
public async Task<int> Count()
{
int i = 0;
var termTexts = await TermTexts;
using (var tx = StateManager.CreateTransaction())
{
var enumerable = await termTexts.CreateEnumerableAsync(tx);
var enumerator = enumerable.GetAsyncEnumerator();
while (await enumerator.MoveNextAsync(CancellationToken.None))
{
i++;
}
}
return i;
}
The total time of this operation is 747165 ms (12 minutes). This is an integer on the wire - no large packets.
My question is, is this performance in the expected region or should I investigate this further? It could be that using a RC for this volume of data is a misuse, so we would need to look elsewhere. These response times imply also that there are disk reads involved, so another question is at what point would this occur and can this be configured? A read from hot storage I would imagine to be in the region below 50ms per partition.
Thanks in advance.

Answer 1

If u enumerate millions of records asynchronously, this performance is expected. If u use IReliableDictionary2, you can use the Count property directly. No need to enumerate all records. If u want to search by key, use CreateAsyncEnumerable function.