Opening many connections with mongodb in a short time causes UncategorizedMongoDbException - mongodb

Question

I deployed 3 java applications on the server. They opened about 60 connections yesterday in one second and throwed MongoSocketOpenException, this is my log:
2019-01-13 13:06:32.121 INFO 3264 --- [qtp1487230025-2553] org.mongodb.driver.connection : Opened connection [connectionId{localValue:19}] to statistic-db:27017
2019-01-13 13:06:32.133 INFO 3264 --- [qtp1487230025-2557] org.mongodb.driver.connection : Opened connection [connectionId{localValue:20}] to statistic-db:27017
2019-01-13 13:06:32.860 INFO 3264 --- [qtp1487230025-2602] org.mongodb.driver.connection : Opened connection [connectionId{localValue:12}] to statistic-db:27017
2019-01-13 13:06:32.860 INFO 3264 --- [qtp1487230025-2522] org.mongodb.driver.connection : Opened connection [connectionId{localValue:13}] to statistic-db:27017
2019-01-13 13:06:33.830 INFO 3264 --- [qtp1487230025-2584] org.mongodb.driver.connection : Closed connection [connectionId{localValue:11}] to statistic-db:27017 because there was a socket exception raised by this connection.
2019-01-13 13:06:33.836 ERROR 3264 --- [qtp1487230025-2584] c.d.controller.AssistanceController : AssistanceRecordController operation pathQuery: /assistance-service/assistance/reward?appId=wx30ad905999f3eea4&activityId=weaponHelp_gun4&targetId=B618CF1A90A43A34B8CC7DA01F7FD4EB&mcachenum=1547355993038 throw exception: class org.springframework.data.mongodb.UncategorizedMongoDbException
message: Exception opening socket; nested exception is com.mongodb.MongoSocketOpenException: Exception opening socket
org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:131)
org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2592)
org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:526)
org.springframework.data.mongodb.core.MongoTemplate.doUpdate(MongoTemplate.java:1405)
org.springframework.data.mongodb.core.MongoTemplate.updateMulti(MongoTemplate.java:1387)
com.diamondcat.service.AssistanceService.consumeAssistance(AssistanceService.java:165)
com.diamondcat.controller.AssistanceController.reward(AssistanceController.java:207)
sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866)
javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.Server.handle(Server.java:531)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
java.lang.Thread.run(Thread.java:748)
From my nginx log, I just know that there are about 50 requests cost more than 2S, maybe have something to do with it.
my java MongoClientOptions is :
#Bean
public MongoClientOptions mongoClientOptions() {
MongoClientOptions.Builder builder = MongoClientOptions.builder();
builder.connectionsPerHost(1000);
builder.threadsAllowedToBlockForConnectionMultiplier(1000);
builder.maxWaitTime(180000);
builder.connectTimeout(2000);
builder.socketTimeout(120000);
builder.writeConcern(new WriteConcern(1, 10000));
return builder.build();
}
My mongodb version is 2.6.
Ijust searched some info, will it be related to the number of files opened by the system? That time is not the highest peak of system access

Answer 1

I just found that there are some very slow query in mongodb. Some query cost more than 400ms, this may cause many connections to be occupied.