I have a pod with 3 containers.
.Net core REST microservice
.Net core Reverse Proxy
Istio proxy
Traffic comes into the Reverse Proxy, is validated, and then proxied onto the microservice. This is my most heavily used service and it starts to have this error after running for about a day.
System.Net.Http.HttpRequestException: Cannot assign requested address
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
Restarting the pod is my only fix right now. I've goofed around looking at tcp statistics on the nodes, but it doesn't make sense that that would be the issue as killing the pod and restarting makes the problem go away.
I've also messed around with the httpclient available in .net core using the best practices with no change.
Any thoughts would be appreciated.
Upon further gathering of clues, I learned that these errors only show up while our REST microservice restarts (due to a memory leak). The error makes sense in context and I overestimated the severity of the issue.
Related
I am running into an intermittent Connection Closed issue resulting in a Cancellation Token.
We have an AKS Cluster with many pods. Each pod has a health check leveraging the AspNetCore Diagnostic HeathCheck pipeline.
We are using Mongo Atlas, hosted in Azure, connecting over a private endpoint.
For each pod's health check, we leverage the AspNetCore Diagnostic Health Check Package for MongoDB. This under the hood, connects to Mongo with a connection string pulled from IConfiguration and connects to the Database with the configuration settings provided by the connection string. It then does a simple query, asking the database to return a list of collections within the database. If this is successful it will return a Healthy Result back from the AspNetCore Diagnostics thus to the K8 probe.
For K8 probe here is the configuration:
livenessProbe:
failureThreshold: 3
httpGet:
path: /Heartbeat
port: 8080
scheme: HTTP
initialDelaySeconds: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
Intermittently we are getting these errors wears the pod reports back an unhealthy connection, saying the connection has been closed. Below is the stack trace.
System.OperationCanceledException: The operation was canceled. at
System.Threading.CancellationToken.ThrowOperationCanceledException() at
MongoDB.Driver.Core.Connections.BinaryConnection.ThrowIfCancelledOrDisposedOrNotOpen(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.SendMessagesAsync(IEnumerable`1 messages, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken) at
MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.HelloHelper.GetResultAsync(IConnection connection, CommandWireProtocol`1 helloProtocol, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.ConnectionInitializer.SendHelloAsync(IConnection connection, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.PooledConnection.OpenAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedInternalAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedOrReuseAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.AcquireConnectionAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.ListCollectionsOperation.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken) at
MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.ExecuteReadOperationAsync[T](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.ListCollectionNamesAsync(IClientSessionHandle session, ListCollectionNamesOptions options, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken) at
HealthChecks.MongoDb.MongoDbHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /_/src/HealthChecks.MongoDb/MongoDbHealthCheck.cs:line 57
I don't see any particular pattern or rhyme or reason why this is occurring especially because it's random. Also it happens when there is little to no traffic on the cluster either. So I have ruled out load.
Any ideas?
Have an Asp.Net Core 2.2.1 Web Api that is trying to obtain OpenId Configuration so that it can authenticate a request. The authority web site happens to be IdentityServer4.
The /.well-known/openid-configuration looks correct and the api site is running and unsecure api methods work ok.
Viewing the logs in Kudu, I see the following error. Appears to be caused by a Sockets issue but I have no idea what's wrong here.
All works fine locally on my dev machine, so what am I missing / is needed for Azure?
System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII is hidden]'. --->
System.Net.Http.HttpRequestException: An attempt was made to access a socket
in a way forbidden by its access permissions --->
Full error detail;
2019-01-11 11:07:43.096 +00:00 [Error] Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler: Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: '[PII is hidden]'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII is hidden]'. ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
After hours of debugging, and switching on
IdentityModelEventSource.ShowPII = true; //show detail of error and see problem
to see the problem with the well-known configuration I saw that it was not picking up application settings from the Azure portal.
As my settings were nested, I just needed to make sure the key in the Azure blade matched.
i.e.
{
"AppSettings": {
"ApiUrl": "someUrl",
"AuthorityUrl": "anotherUrl"
}, ...
becomes
AppSettings:ApiUrl
Simple mistake gotcha.
It is working in my local code .It is not working in my published website:
System.Net.Mail.SmtpException: Failure sending mail. --->
System.Net.WebException: Unable to connect to the remote server --->
System.Net.Sockets.SocketException: An attempt was made to access a
socket in a way forbidden by its access permissions 40.97.144.50:587
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot,
SocketAddress socketAddress) at
System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,
Socket s4, Socket s6, Socket& socket, IPAddress& address,
ConnectSocketState state, IAsyncResult asyncResult, Exception&
exception) --- End of inner exception stack trace --- at
System.Net.ServicePoint.GetConnection(PooledStream PooledStream,
Object owner, Boolean async, IPAddress& address, Socket& abortSocket,
Socket& abortSocket6) at System.Net.PooledStream.Activate(Object
owningObject, Boolean async, GeneralAsyncDelegate asyncCallback) at
System.Net.PooledStream.Activate(Object owningObject,
GeneralAsyncDelegate asyncCallback) at
System.Net.ConnectionPool.GetConnection(Object owningObject,
GeneralAsyncDelegate asyncCallback, Int32 creationTimeout) at
System.Net.Mail.SmtpConnection.GetConnection(ServicePoint
servicePoint) at
System.Net.Mail.SmtpTransport.GetConnection(ServicePoint servicePoint)
at System.Net.Mail.SmtpClient.GetConnection() at
System.Net.Mail.SmtpClient.Send(MailMessage message) --- End of inner
exception stack trace ---
Kindly suggest some solutions
I get the following exception thrown occasionally when trying to write documents to my collection.
I am using MongoDB 3.0.7 hosted in mLabs. Our application is hosted in Azure (as a Web App) and I am using the C# 2.2.3 SDK.
MongoDB.Driver.MongoConnectionException: An exception occurred while
opening a connection to the server. --->
System.Net.Sockets.SocketException: An attempt was made to access a
socket in a way forbidden by its access permissions
191.235.xxx.xxx:54128 at System.Net.Sockets.Socket.Connect(IPAddress[] addresses, Int32 port)
at System.Net.Sockets.Socket.Connect(String host, Int32 port) at
MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket
socket, EndPoint endPoint, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint
endPoint, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken
cancellationToken) --- End of inner exception stack trace --- at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.Open(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Servers.ClusterableServer.GetChannel(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding
binding, CancellationToken cancellationToken) at
MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding
binding, IWriteOperation'1 operation, CancellationToken
cancellationToken) at
MongoDB.Driver.MongoCollectionImpl'1.ExecuteWriteOperation[TResult](IWriteOperation`1
operation, CancellationToken cancellationToken) at
MongoDB.Driver.MongoCollectionImpl'1.BulkWrite(IEnumerable'1 requests,
BulkWriteOptions options, CancellationToken cancellationToken) at
MongoDB.Driver.MongoCollectionBase'1.InsertOne(TDocument document,
InsertOneOptions options, CancellationToken cancellationToken)
This happens if you hit the sandbox limits. More details in my old answer Intermittent crashes in Azure Web Application
To know about sandbox limits please refer to https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox
Also came across http://www.freekpaans.nl/2015/08/starving-outgoing-connections-on-windows-azure-web-sites/ which was nicely written on the same subject
Edit - February 2018
If you are running in to this issue now, go to Diagnose and Solve Blade for your App in the Azure portal and check the TCP Connections tile. It should give you information about the site consuming the maximum ports and allow you to monitor the state connections instance wide.
I am facing an issue with my socket connection. I have a web Service in Project and It connect to a service installed in the same server. The Service is getting crashed and it has successfully restarted by another monitoring application in our project. But when the web method tries to access the method in the service I am getting a web exception. But when we restart the service manually it works fine. Here is web exception I am getting and Code used.
Code Using
string url = "http://localhost:56789/" + SERVICE_NAME;
m_target = Activator.GetObject(typeof(IWebServiceInterface), url) as IWebServiceInterface;
if (IPAddress.TryParse(ipAddress, out ip))
{
status = m_target.Connect(Session.SessionID, username, stationId, ip);
}
Web Exception
Exception rethrown at [0]: at
System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg) at
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at
TerminalInterface.IWebServiceInterface.Connect(String sessionId,
String username, Int32 stationId, IPAddress ipAddress) at
myWebProject.WebServices.Terminal.Service.Connect(String username,
Int32 stationId, String ipAddress)
18:42:28.953 ProLog: System.Net.WebException: Unable to connect to the
remote server ---> System.Net.Sockets.SocketException: No connection
could be made because the target machine actively refused it
127.0.0.1:56789 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at
System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP) at
System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,
Socket s4, Socket s6, Socket& socket, IPAddress& address,
ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout,
Exception& exception) --- End of inner exception stack trace ---
Server stack trace: at
System.Net.HttpWebRequest.GetRequestStream() at
System.Runtime.Remoting.Channels.Http.HttpClientTransportSink.ProcessAndSend(IMessage
msg, ITransportHeaders headers, Stream inputStream) at
System.Runtime.Remoting.Channels.Http.HttpClientTransportSink.ProcessMessage(IMessage
msg, ITransportHeaders requestHeaders, Stream requestStream,
ITransportHeaders& responseHeaders, Stream& responseStream) at
System.Runtime.Remoting.Channels.SoapClientFormatterSink.SyncProcessMessage(IMessage
msg)