Azure Web Api Jwt - Unable to obtain configuration Socket forbidden - jwt

Have an Asp.Net Core 2.2.1 Web Api that is trying to obtain OpenId Configuration so that it can authenticate a request. The authority web site happens to be IdentityServer4.
The /.well-known/openid-configuration looks correct and the api site is running and unsecure api methods work ok.
Viewing the logs in Kudu, I see the following error. Appears to be caused by a Sockets issue but I have no idea what's wrong here.
All works fine locally on my dev machine, so what am I missing / is needed for Azure?
System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII is hidden]'. --->
System.Net.Http.HttpRequestException: An attempt was made to access a socket
in a way forbidden by its access permissions --->
Full error detail;
2019-01-11 11:07:43.096 +00:00 [Error] Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler: Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: '[PII is hidden]'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII is hidden]'. ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
--- End of inner exception stack trace ---
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()

After hours of debugging, and switching on
IdentityModelEventSource.ShowPII = true; //show detail of error and see problem
to see the problem with the well-known configuration I saw that it was not picking up application settings from the Azure portal.
As my settings were nested, I just needed to make sure the key in the Azure blade matched.
i.e.
{
"AppSettings": {
"ApiUrl": "someUrl",
"AuthorityUrl": "anotherUrl"
}, ...
becomes
AppSettings:ApiUrl
Simple mistake gotcha.

Related

MongoDB HealthCheck Intermittently getting CancellationToken

I am running into an intermittent Connection Closed issue resulting in a Cancellation Token.
We have an AKS Cluster with many pods. Each pod has a health check leveraging the AspNetCore Diagnostic HeathCheck pipeline.
We are using Mongo Atlas, hosted in Azure, connecting over a private endpoint.
For each pod's health check, we leverage the AspNetCore Diagnostic Health Check Package for MongoDB. This under the hood, connects to Mongo with a connection string pulled from IConfiguration and connects to the Database with the configuration settings provided by the connection string. It then does a simple query, asking the database to return a list of collections within the database. If this is successful it will return a Healthy Result back from the AspNetCore Diagnostics thus to the K8 probe.
For K8 probe here is the configuration:
livenessProbe:
failureThreshold: 3
httpGet:
path: /Heartbeat
port: 8080
scheme: HTTP
initialDelaySeconds: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
Intermittently we are getting these errors wears the pod reports back an unhealthy connection, saying the connection has been closed. Below is the stack trace.
System.OperationCanceledException: The operation was canceled. at
System.Threading.CancellationToken.ThrowOperationCanceledException() at
MongoDB.Driver.Core.Connections.BinaryConnection.ThrowIfCancelledOrDisposedOrNotOpen(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.SendMessagesAsync(IEnumerable`1 messages, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken) at
MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.HelloHelper.GetResultAsync(IConnection connection, CommandWireProtocol`1 helloProtocol, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.ConnectionInitializer.SendHelloAsync(IConnection connection, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.PooledConnection.OpenAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedInternalAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedOrReuseAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.AcquireConnectionAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Operations.ListCollectionsOperation.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken) at
MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.ExecuteReadOperationAsync[T](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.ListCollectionNamesAsync(IClientSessionHandle session, ListCollectionNamesOptions options, CancellationToken cancellationToken) at
MongoDB.Driver.MongoDatabaseImpl.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken) at
HealthChecks.MongoDb.MongoDbHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /_/src/HealthChecks.MongoDb/MongoDbHealthCheck.cs:line 57
I don't see any particular pattern or rhyme or reason why this is occurring especially because it's random. Also it happens when there is little to no traffic on the cluster either. So I have ruled out load.
Any ideas?

Kubernetes inter-pod port exhaustion

I have a pod with 3 containers.
.Net core REST microservice
.Net core Reverse Proxy
Istio proxy
Traffic comes into the Reverse Proxy, is validated, and then proxied onto the microservice. This is my most heavily used service and it starts to have this error after running for about a day.
System.Net.Http.HttpRequestException: Cannot assign requested address
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
Restarting the pod is my only fix right now. I've goofed around looking at tcp statistics on the nodes, but it doesn't make sense that that would be the issue as killing the pod and restarting makes the problem go away.
I've also messed around with the httpclient available in .net core using the best practices with no change.
Any thoughts would be appreciated.
Upon further gathering of clues, I learned that these errors only show up while our REST microservice restarts (due to a memory leak). The error makes sense in context and I overestimated the severity of the issue.

Error sending message to Mule API from BizTalk

I am trying to post a message to Mule service, which is expected to respond with Success or Error Response. When I post a request directly to the Mule API using Postman, it gives a response as 200 with some HTML content. When I am pushing the same request with similar headers from BizTalk send port, I get the following error:
An error occurred while processing the message, refer to the details section for more information Message ID: {56AF4477-590F-415E-9D70-536692FE637C} Instance ID: {20EEC344-3B1A-474E-ACEB-160CBF616775} Error Description: System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to https://. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.Connection.ReadCallback(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result) --- End of inner exception stack trace --- Server stack trace: at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndRequest(IAsyncResult result) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at System.ServiceModel.Channels.IRequestChannel.EndRequest(IAsyncResult result) at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient2.RequestCallback(IAsyncResult result)
This call from the BizTalk interface to Mule API was working fine until yesterday.
I have tried using Fiddler to see the HTTP traffic from the BizTalk server. It shows the handshake with the Mule service, but no further information on headers or the request body.
Please help.
UPDATE
When I posted the request directly to the Mule Api from another machine, it gave a proper response. This seems like a network issue, some rules may have been modified to allow traffic from specific servers. Waiting for the network team to confirm.

Intermittent MongoDB Connection Issue: An attempt was made to access a socket in a way forbidden by its access permissions

I get the following exception thrown occasionally when trying to write documents to my collection.
I am using MongoDB 3.0.7 hosted in mLabs. Our application is hosted in Azure (as a Web App) and I am using the C# 2.2.3 SDK.
MongoDB.Driver.MongoConnectionException: An exception occurred while
opening a connection to the server. --->
System.Net.Sockets.SocketException: An attempt was made to access a
socket in a way forbidden by its access permissions
191.235.xxx.xxx:54128 at System.Net.Sockets.Socket.Connect(IPAddress[] addresses, Int32 port)
at System.Net.Sockets.Socket.Connect(String host, Int32 port) at
MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket
socket, EndPoint endPoint, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint
endPoint, CancellationToken cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken
cancellationToken) --- End of inner exception stack trace --- at
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.Open(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Servers.ClusterableServer.GetChannel(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken
cancellationToken) at
MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding
binding, CancellationToken cancellationToken) at
MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding
binding, IWriteOperation'1 operation, CancellationToken
cancellationToken) at
MongoDB.Driver.MongoCollectionImpl'1.ExecuteWriteOperation[TResult](IWriteOperation`1
operation, CancellationToken cancellationToken) at
MongoDB.Driver.MongoCollectionImpl'1.BulkWrite(IEnumerable'1 requests,
BulkWriteOptions options, CancellationToken cancellationToken) at
MongoDB.Driver.MongoCollectionBase'1.InsertOne(TDocument document,
InsertOneOptions options, CancellationToken cancellationToken)
This happens if you hit the sandbox limits. More details in my old answer Intermittent crashes in Azure Web Application
To know about sandbox limits please refer to https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox
Also came across http://www.freekpaans.nl/2015/08/starving-outgoing-connections-on-windows-azure-web-sites/ which was nicely written on the same subject
Edit - February 2018
If you are running in to this issue now, go to Diagnose and Solve Blade for your App in the Azure portal and check the TCP Connections tile. It should give you information about the site consuming the maximum ports and allow you to monitor the state connections instance wide.

No connection could be made because the target machine actively refused it 127.0.0.1:56789

I am facing an issue with my socket connection. I have a web Service in Project and It connect to a service installed in the same server. The Service is getting crashed and it has successfully restarted by another monitoring application in our project. But when the web method tries to access the method in the service I am getting a web exception. But when we restart the service manually it works fine. Here is web exception I am getting and Code used.
Code Using
string url = "http://localhost:56789/" + SERVICE_NAME;
m_target = Activator.GetObject(typeof(IWebServiceInterface), url) as IWebServiceInterface;
if (IPAddress.TryParse(ipAddress, out ip))
{
status = m_target.Connect(Session.SessionID, username, stationId, ip);
}
Web Exception
Exception rethrown at [0]: at
System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg) at
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at
TerminalInterface.IWebServiceInterface.Connect(String sessionId,
String username, Int32 stationId, IPAddress ipAddress) at
myWebProject.WebServices.Terminal.Service.Connect(String username,
Int32 stationId, String ipAddress)
18:42:28.953 ProLog: System.Net.WebException: Unable to connect to the
remote server ---> System.Net.Sockets.SocketException: No connection
could be made because the target machine actively refused it
127.0.0.1:56789 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at
System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP) at
System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,
Socket s4, Socket s6, Socket& socket, IPAddress& address,
ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout,
Exception& exception) --- End of inner exception stack trace ---
Server stack trace: at
System.Net.HttpWebRequest.GetRequestStream() at
System.Runtime.Remoting.Channels.Http.HttpClientTransportSink.ProcessAndSend(IMessage
msg, ITransportHeaders headers, Stream inputStream) at
System.Runtime.Remoting.Channels.Http.HttpClientTransportSink.ProcessMessage(IMessage
msg, ITransportHeaders requestHeaders, Stream requestStream,
ITransportHeaders& responseHeaders, Stream& responseStream) at
System.Runtime.Remoting.Channels.SoapClientFormatterSink.SyncProcessMessage(IMessage
msg)