I have a single primary instance of a MongoDB to connect to. I am using the C# driver (latest revision) to connect and utilise MongoDB.
My client application is multi-threaded whereby processes will run periodically at different intervals to produce reports. When more than one thread is running, and I call various functions on the driver (e.g server.DatabaseExists(...)), I am consistently met with an exception.
"The requested name is valid, but no data of the requested type was found"
at System.Net.Dns.GetAddrInfo(String name)
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
at MongoDB.Driver.MongoServerAddress.ToIPEndPoint(AddressFamily addressFamily) in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoServerAddress.cs:line 195
at MongoDB.Driver.MongoServerInstance.GetIPEndPoint() in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoServerInstance.cs:line 339
at MongoDB.Driver.Internal.MongoConnection.Open() in C:\work\rstam\mongo-csharp-driver\Driver\Internal\MongoConnection.cs:line 368
at MongoDB.Driver.Internal.MongoConnection.GetNetworkStream() in C:\work\rstam\mongo-csharp-driver\Driver\Internal\MongoConnection.cs:line 574
at MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoRequestMessage message, SafeMode safeMode, String databaseName) in C:\work\rstam\mongo-csharp-driver\Driver\Internal\MongoConnection.cs:line 555
at MongoDB.Driver.MongoCursorEnumerator`1.GetReply(MongoConnection connection, MongoRequestMessage message) in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator.cs:line 295
at MongoDB.Driver.MongoCursorEnumerator`1.GetFirst() in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator.cs:line 253
at MongoDB.Driver.MongoCursorEnumerator`1.MoveNext() in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator.cs:line 141
at MongoDB.Driver.MongoDatabase.GetCollectionNames() in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoDatabase.cs:line 645
at MongoDB.Driver.MongoDatabase.CollectionExists(String collectionName) in C:\work\rstam\mongo-csharp-driver\Driver\Core\MongoDatabase.cs:line 297
Running one thread does not cause this issue. I am of the understanding that MongoDB is thread-safe, so I am at loss to understand the cause and the remedy.
The only "solution" I've found was to recycle the Application Pool that's hosting my website\service calling the mongo client.
Related
I am trying to implement the GitHub project (https://github.com/tomatoTomahto/CDH-Sensor-Analytics) on our internal Hadoop cluster via Cloudera Data Science Workbench.
On running the project on Cloudera Data Science Workbench, I get the error "No Brokers available" when trying to connect to Kafka through Python api KafkaProducer(bootstrap_servers='broker1:9092') [Code can be found in https://github.com/tomatoTomahto/CDH-Sensor-Analytics/blob/master/datagenerator/KafkaConnection.py].
I have authenticated using Kerberos. I have tried giving broker node without port number, and also as a list. But, nothing has worked so far.
Below is the stack trace.
NoBrokersAvailable: NoBrokersAvailable
NoBrokersAvailable Traceback (most recent call
last)
in engine
----> 1 dgen = DataGenerator(config)
/home/cdsw/datagenerator/DataGenerator.py in __init__(self, config)
39
40 self._kudu = KuduConnection(self._config['kudu_master'],
self._config['kudu_port'], spark)
---> 41 self._kafka =
KafkaConnection(self._config['kafka_brokers'],
self._config['kafka_topic'])
42
43 #self._kafka
/home/cdsw/datagenerator/KafkaConnection.py in __init__(self, brokers,
topic)
4 class KafkaConnection():
5 def __init__(self, brokers, topic):
----> 6 self._kafka_producer =
KafkaProducer(bootstrap_servers=brokers)
7 self._topic = topic
8
/home/cdsw/.local/lib/python3.6/site-packages/kafka/producer/kafka.py
in __init__(self, **configs)
333
334 client = KafkaClient(metrics=self._metrics,
metric_group_prefix='producer',
--> 335 **self.config)
336
337 # Get auto-discovered version from client if necessary
/home/cdsw/.local/lib/python3.6/site-packages/kafka/client_async.py in
__init__(self, **configs)
208 if self.config['api_version'] is None:
209 check_timeout =
self.config['api_version_auto_timeout_ms'] / 1000
--> 210 self.config['api_version'] =
self.check_version(timeout=check_timeout)
211
212 def _bootstrap(self, hosts):
/home/cdsw/.local/lib/python3.6/site-packages/kafka/client_async.py in
check_version(self, node_id, timeout, strict)
806 try_node = node_id or self.least_loaded_node()
807 if try_node is None:
--> 808 raise Errors.NoBrokersAvailable()
809 self._maybe_connect(try_node)
810 conn = self._conns[try_node]
NoBrokersAvailable: NoBrokersAvailable
I also tried connecting outside of workbench through CLI by having VPN connection. I got the same error.
Any pointers on what am I missing? Thanks in advance!
The first step is establishing whether the network route is open and the broker is up and listening on that port. After that you can check authentication, etc.
Did you try telnet <broker host> 9092?
You may need to explicitly set advertised.listeners in addition to listeners, I've occasionally seen a weird quirk with Java where it wasn't binding to the expected network interface (or at least the one I expected!) and I had to force it using advertised.listeners.
I'm experiencing intermittent connectivity errors from a Spring Boot application communicating with a D1 Google CloudSQL Server with the configuration settings described here HikariCP MySQL settings
I was wondering if anyone has encountered this before.
I've read the FAQ posted here Hikari FAQ and I'm wondering if my default idleTimeout and maxLifeTime (30 mins) settings might be at fault; wait_timeout and interactive_timeout on the server are both set to default 28800s (8 hours).
The FAQ says that these two settings should be about a minute less that the server settings, but if I'm losing connections after 30 minutes I can't quite see how upping the maxLifeTime to 7hrs 59mins is going to improve the situation.
Does anyone have any recommendations?
Redacted stack trace(s):
Get these from time to time
org.springframework.security.authentication.InternalAuthenticationServiceException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Timeout after 30018ms of waiting for a connection.
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:110)
at org.springframework.security.authentication.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:132)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:177)
...
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Timeout after 30023ms of waiting for a connection.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
....
Caused by: java.sql.SQLException: Timeout after 30023ms of waiting for a connection.
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:208)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:108)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 59 common frames omitted
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:737)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:787)
Hibernate search:
2015-02-17 10:34:17.090 INFO 1 --- [ entityloader-2] o.h.s.i.SimpleIndexingProgressMonitor : HSEARCH000030: 31050 documents indexed in 1147865 ms
2015-02-17 10:34:17.090 INFO 1 --- [ entityloader-2] o.h.s.i.SimpleIndexingProgressMonitor : HSEARCH000031: Indexing speed: 27.050219 documents/second; progress: 99.89%
2015-02-17 10:41:59.917 WARN 1 --- [ntifierloader-1] com.zaxxer.hikari.proxy.ConnectionProxy : Connection com.mysql.jdbc.JDBC4Connection#372f2018 (HikariPool-0) marked as broken because of SQLSTATE(08S01), ErrorCode(0).
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,611,087 milliseconds ago. The last packet sent successfully to the server was 927,899 milliseconds ago.
The indexing isn't particularly quick at the moment I think because I'm not using projections. The process takes about 30 minutes to execute.
Thanks
It could be a couple of things here. First, the network infrastructure (firewalls, load-balancers, etc.) between the application tier and the database tier can impose their own connection timeouts, regardless of MySql settings.
The indexing failure indicates that the connection was out of the pool for ~27 minutes with no SQL activity when that failure occurred.
Second, specifically regarding the "Could not get JDBC Connection" error, you may be running into Cloud SQL connection limits.
I recommend three things. One, make sure you are on the latest HikariCP (2.3.2) and latest MySql Connector/J driver (5.1.34). Two, enable DEBUG-level logging for the com.zaxxer.hikari package. HikariCP debug logging is not "chatty", but will log pool statistics every 30 seconds (and sometimes more detail in failure conditions). Lastly, try setting the maxPoolSize to something smaller (unless already at the default), and setting maxLifeTime to 15 or 20 minutes (1200000ms).
If the error occurs again, post updated logs containing the HikariCP debug logs around the time of failure. Also, feel free to open a tracking issue over on Github as larger logs etc. are easier there.
Using MongoDB 2.4.5 with version 0.702 of the Perl MongoDB driver, I frequently run into exceptions like these:
recv timed out (30000 ms) at ...MongoDB/Cursor.pm line 161.
couldn't get response to throw out at ...MongoDB/Cursor.pm line 161.
missed the response we wanted, please try again at ...MongoDB/Cursor.pm line 161
invalid header received at ...MongoDB/Cursor.pm line 161.
can't get db response, not connected at ...MongoDB/Cursor.pm line 161.
The exceptions are intermittent, and often vanish on the next request (this is a web app). Occasionally, the exceptions will persist over several consecutive requests.
This is a tiny database running the default configuration (no sharding or anything fancy). I've tried using some of the tools listed here and here, but I'm unclear on how to apply them to this situation.
This is all running on Debian 7.1 64-bit. The web server is Mojolicious' hypnotoad 4.07 on perl 5.16.3 running behind apache2.
Can you kindly suggest some tools & strategies for diagnosing the problem? Thanks for your time.
When using access 2003 to convert an access 2000 database to access 97 I get the following 2 error messages right after the other
'invalid argument'
'Some errors happened during the conversion. No converted database was generated.'
Other than this there is no information provided.
Has anyone come across this before and have any clues about how to find the error
I compacted/repaired the database but this had no effect.
This error occurred because on conversion the size of the Access 97 database would have exceeded 1 GB which is the limit for that version of Access.
I am executing below C# code -
for (; ; )
{
Console.WriteLine("Doc# {0}", ctr++);
BsonDocument log = new BsonDocument();
log["type"] = "auth";
BsonDateTime time = new BsonDateTime(DateTime.Now);
log["when"] = time;
log["user"] = "staticString";
BsonBoolean bol = BsonBoolean.False;
log["res"] = bol;
coll.Insert(log);
}
When I run it on a MongoDB instance (version 2.0.2) running on virtual 64 bit Linux machine with just 512 MB ram, I get about 5k inserts with 1-2 faults as reported by mongostat after few mins.
When same code is run against a MongoDB instance (version 2.0.2) running on a physical Windows machine with 8 GB of ram, I get 2.5k inserts with about 80 faults as reported by mongostat after few mins.
Why more faults are occurring on Windows? I can see following message in logs-
[DataFileSync] FlushViewOfFile failed 33 file
Journaling is disable on both instances
Also, is 5k insert on a virtual machine with 1-2 faults a good enough speed? or should I be expecting better inserts?
Looks like this is a known issue - https://jira.mongodb.org/browse/SERVER-1163
page fault counter on Windows is in fact the total page faults which include both hard and soft page fault.
Process : Page Faults/sec. This is an indication of the number of page faults that
occurred due to requests from this particular process. Excessive page faults from a
particular process are an indication usually of bad coding practices. Either the
functions and DLLs are not organized correctly, or the data set that the application
is using is being called in a less than efficient manner.