A connection attempt failed error when accessing MongoDB under load test - mongodb

Question

We're beginning to leverage MongoDB for search functionality in our application, to improve performance over how we used to do searches with SQL Server. Individual testing works fine, however now we've begun load testing with 40 concurrent "users" executing searches against MongoDB at once. Part way through each load test, many of the calls start erroring:
MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.AggregateOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
at MongoDB.Driver.OperationExecutor.ExecuteReadOperation[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.Aggregate[TResult](IClientSessionHandle session, PipelineDefinition`2 pipeline, AggregateOptions options, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass19_0`1.<Aggregate>b__0(IClientSessionHandle session)
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.Aggregate[TResult](PipelineDefinition`2 pipeline, AggregateOptions options, CancellationToken cancellationToken)
at RealtorsPropertyResource.Rpr.MongoSearches.Model.QueryPropertiesForKeys[T](PerformanceMonitor performanceMonitor, PropertiesSearchCriteria criteria, RequiredDataResult requiredData)
at RealtorsPropertyResource.Rpr.MongoSearches.Model.SearchWithMongo(PropertiesSearchCriteria criteria, PerformanceMonitor performanceMonitor)
at RealtorsPropertyResource.Rpr.MongoSearches.Model.SearchForProperties(PropertiesSearchCriteria criteria, PerformanceMonitor performanceMonitor, Nullable`1 useMongo)
at RealtorsPropertyResource.Rpr.Websites.Main.AjaxService.ExecuteLoadTestSearch(AppPropertyMode appPropertyMode, SearchType searchType, String query) in C:\BuildAgents\RPR-Website\_work\4\s\Websites\Main\services\AjaxService.Search.cs:line 260
at SyncInvokeExecuteLoadTestSearch(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
We tested against two Mongo instances, one running on Windows and one running on Linux, and both are hitting this error. The searches that are completing successfully aren't slow, so I'm not confident this is a timeout issue, but perhaps maybe a max connection issue? I'm confused though because I thought the default connection limit for the MongoDB C# driver was 100 connections, and that limit for the MongoDB server itself was much higher than that.
Any suggestions of how to troubleshoot this error that only occurs while Mongo is under load?
In case it's relevant, here is how I'm instantiating a singleton MongoClient and IMongoDatabase:
public static class MongoContext
{
private static bool initialized = false;
private static string connectionString { get; set; }
private static IMongoDatabase database;
public static IMongoDatabase Database
{
get
{
if (database == null)
throw new InvalidOperationException("You must initialize the MongoContext first");
return database;
}
}
public static void Initialize()
{
if (!initialized)
{
initialized = true;
connectionString = RprConfigurationManager.GetConnectionString("MongoSearch");
Console.WriteLine("Connecting to MongoDB search database: " + connectionString);
var useSsl = connectionString.IndexOf("ssl=true", StringComparison.InvariantCultureIgnoreCase) >= 0;
var url = new MongoUrl(connectionString);
var clientSettings = MongoClientSettings.FromUrl(url);
clientSettings.SslSettings = new SslSettings();
clientSettings.SslSettings.CheckCertificateRevocation = false;
clientSettings.UseSsl = useSsl;
clientSettings.VerifySslCertificate = false;
var client = new MongoClient(clientSettings);
var lastSegment = connectionString.Split('/').Last();
var databaseSegments = lastSegment.Split('?');
var databaseName = databaseSegments.First();
databaseName = !String.IsNullOrEmpty(databaseName) ? databaseName : "rpr";
Console.WriteLine("Connecting to database: " + databaseName);
database = client.GetDatabase(databaseName);
SchemaInitializer.RegisterClassMaps();
}
}
}
Here are some mongod logs, in case it helps:
IXSCAN { CityPlaceId: 1, ForSaleGroupId: 1, DistressedGroupId: 1,
OffMarketGroupId: 1, ForLeaseGroupId: 1, ForSaleDistressedGroupId: 1,
OffMarketDistressedGroupId: 1, OrgId: 1, OrgSecurableId: 1,
OffMarketDate: 1, IsListingOnly: 1 } keysExamined:397707
docsExamined:332738 hasSortStage:1 cursorExhausted:1 numYields:20750
nreturned:1 reslen:1454 locks:{ Global: { acquireCount: { r: 21121 }
}, Database: { acquireCount: { r: 21121 } }, Collection: {
acquireCount: { r: 21121 } } } storage:{ data: { bytesRead:
7534883557, timeReadingMicros: 325960940 } } protocol:op_msg 356835ms
2019-08-19T19:43:08.856+0000 I NETWORK [conn285] Error sending
response to client: SocketException: An established connection was
aborted by the software in your host machine.. Ending connection from
172.27.129.209:65121 (connection id: 285) 2019-08-19T19:43:08.856+0000 I NETWORK [conn285] end connection 172.27.129.209:65121 (30
connections now open)

Answer 1

Here is mentioned that can be a problem with the driver, In our case we change the driver version from 2.5 to 2.10 and worked.