We lift and shifted our cloud services to Azure Service Fabric. We converted our services to Windows Container and successfully deployed to the cluster. However, we're getting a lot of timeout issue while uploading file to Sharepoint and accessing Azure SQL Server.
See the example below:
SQL: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
Sharepoint: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
Note: These issues are not happening is cloud services.
Related
I have an elastic apm server version 7.17.1. There are only to Django application on the server. The APM service is using about 140MB of memory. When connecting new agents, I receive a timeout error.
node.js error
{"log.level":"error","#timestamp":"2022-07-07T23:15:34.033Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error: intake response timeout: APM server did not respond within 10s of gzip stream finish"}
Java error
2022-07-07 14:31:36,503 [elastic-apm-server-reporter] ERROR
co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error
sending data to APM server: Read timed out, response code is -1
If I use flask or Django or PHP, new apps are registered.
(I couldn't find logs from Go, but that APM agent failed as well.)
APM server logs did not appear relevant as the errors occurred in both instances
Check your IPS. In our env, we saw the IPS swatting down packets from Java and Go-based apps. I'm no developer, but once we got those sources white-listed, we saw the Agents information coming through :)
When trying to connect to AWS DocumentDB using mongocxx C++ driver, even after using the AWS combined pem file as a URI parameter (CA file), I get the below error of TLS handshake failed.
No suitable servers found (`serverSelectionTryOnce` set): [TLS handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed calling ismaster on 'docdb-xxxxxxxx.ap-southeast-1.docdb.amazonaws.com:27017']: generic server error
I have masked out the full hostname of the documentdb instance. I am using the connection URI method mentioned in http://mongocxx.org/mongocxx-v3/configuration/
// 2) Using the URI
auto client2 = mongocxx::client{uri{"mongodb://host1/?tls=true&tlsAllowInvalidCertificates=true&tlsCAFile=/path/to/custom/cert.pem"}};
I am using mongocxx 3.4.2 and libmongoc 1.16.2
I have tried this connection with the Node.js driver and it is able to connection. Any ideas what could be wrong?
I was trying to connect to my DocumentDB cluster on AWS via an external app like TablePlus and I had the same error:
No suitable servers found
(`serverSelectionTryOnce` set): [Failed to resolve 'docdb-1984-08-10-12-14-15.cluster-boogeyman.xy-central-99.docdb.amazonaws.com']
What I tried next is to:
open all sort of incoming traffic in the security group assigned to my cluster
made sure that "Encryption-at-rest" (in Advanced Settings) is disabled while creating the cluster
I still got this error. What I discovered next is that:
Trying to connect to an Amazon DocumentDB cluster directly from a public endpoint, such as your laptop or local development machine, will fail. Amazon DocumentDB is virtual private cloud (VPC)-only and does not currently support public endpoints. Thus, you can't connect directly to your Amazon DocumentDB cluster from your laptop or local development environment outside of your VPC.
Please read the AWS connection troubleshooting section. To connect to an Amazon DocumentDB cluster from outside an Amazon VPC, you can use an SSH tunnel.
I have configured Bluemix Secure gateway client to connect to DB2 on AS 400 using a Windows machine. I am able to run the Db2 connection successfully from the application on Windows machine where Secure gateway client is running. But when I run the same application on Bluemix with modifying the code to depict Secure Gateway client destination details, I get 'Connection Refused' error.
I had gone through how to connect a bluemix app to on-premise db/as400? , which mentions about Secure gateway client for OS 400.
Wanted to understand how to fix the issue for 'Connection Refused' ? Do we have to use different DB2Driver/ DB2 Connect string from Bluemix? or Any other settings to be done on AS400 server?
Given that the Connection refused error is occurring between your BlueMix application and the SG Servers and no logs are being generated on the SG Client, it sounds like the issue is with a listener not being opened on the SG Servers rather than an issue between the SG Client and DB2. All listeners should come up when the SG Client initially connects to the SG Servers.
Are you able to cURL or telnet the cloud host:port provided with this particular destination? Or is this connection refusal specific to your BlueMix app? In either case, this answer will provide more direction on where to investigate further.
I have some timeout problems when connecting my Azure Web App to a MongoDb hosted on a Azure VM.
2015-12-19T15:57:47.330+0100 I NETWORK Socket recv() errno:10060 A connection attempt
failed because the connected party did not properly respond after a period of time,
or established connection failed because connected host has failed to respond.
2015-12-19T15:57:47.343+0100 I NETWORK SocketException: remote: 104.45.x.x:27017 error:
9001 socket exception [RECV_ERROR] server [104.45.x.x:27017]
2015-12-19T15:57:47.350+0100 I NETWORK DBClientCursor::init call() failed
Currently mongodb is configured on a single server (just for dev) and it is exposed through a public ip. Website connect to it using an azure domain name (*.westeurope.cloudapp.azure.com) and without a Virtual Network.
Usually everything works well, but after some minutes of inactivity I get that timeout exception. The same will happen when using the MongoDb shell from my PC, so I'm quite sure that it is a problem on mongodb side.
I'm missing some configuration?
After some searching here my considerations:
It is usually a good practice to implement some sort of retry logic on every resource that you access on Azure (database, VM, ...). For MongoDb there is a partial implementation so you should potentially write your own. See also this issue and this.
If possible all resources on Azure should be in the same Azure Virtual Network (in this way all connections are made using Azure Private Ip instead of Public Ip. This is also useful for security reasons because you don't need to open endpoint to the public.
When deploying MongoDb on Azure try to follow the official MongoDb guidelines.
In this particular case you should set the net.ipv4.tcp_keepalive_time to a value lower than the tcp keep alive of Azure, that by default is 240 seconds. In this way the connection is closed and MongoDb driver can intercept this condition and open a new connection. If the connection is closed by Azure the driver cannot intercept it. If you want to change this setting on Azure (not recommended) you can find it inside the Public Ip configuration.
In my development environment I have set the net.ipv4.tcp_keepalive_time to 120 and now everything seems to work fine. Consider that if you host MondoDb inside an Docker container you should set this setting on the Docker host.
Here some other useful links:
http://focusmatic.tumblr.com/post/39569711018/solving-mongodb-connection-losses-on-windows-azure
https://docs.mongodb.org/ecosystem/platforms/windows-azure/
https://michaelmckeownblog.wordpress.com/2013/12/04/resolving-internal-ips-vs-dns-names-between-vms/
https://gist.github.com/davideicardi/f2094c4c3f3e00fbd490
MongoDB connection problems on Azure
MongoDB connection timeouts (Azure)
When using the C# Mongo driver we resolved this by setting the following
MongoDefaults.MaxConnectionIdleTime = TimeSpan.FromMinutes(1);
We are noticing a number of random failures on the spring cloud services that we created trying to connect to the discovery server. They seem to happen every now and then and so far have not caused discovery to fail.
However, out operations group is reporting this to us a possible issue and we are trying to investigate. Is there any condition that we should be concerned about.
The example that I am showing below is that i replicated the same issue running on my laptop with the service and discovery server (eureka) on the same machine, so networking as a cause seems to be not the issue.
015-08-10 10:41:51.592 INFO 9897 --- [pool-8-thread-1] o.a.http.impl.client.DefaultHttpClient : I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://localhost:8761: The target server failed to respond
2015-08-10 10:41:51.593 INFO 9897 --- [pool-8-thread-1] o.a.http.impl.client.DefaultHttpClient : Retrying request to {}->http://localhost:8761