I' having issues with Azure SignalR Service... Often (several times an hours) its slow and sometimes it even times out...
02 Jan 2019 11:37:47.1172019-01-02 14:37:47.0436 | ERROR | WebPortal | Error
while trying to send message. ConnectionState: Disconnected Channel: Document eventName: DocumentAccess message: Core.DTOs.Documents.RealTime.DocumentRealTimeViewModel - Exception: System.TimeoutException: Server timeout (10000.00ms) elapsed without receiving a message from the server.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsyncCore>d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at RealTimeServices.Services.RealTimeService.<SendMessageAsync>d__9.MoveNext() in D:\home\site\repository\Dms\Services\RealTimeServices\Services\RealTimeService.cs:line 80; LogSource: RealTimeService
Even when doing the negotiating between the SignalR webapp and the Azure SignalR service its slow..
Any help is appreciated...
Related
Azure Cosmos DB containers must have a partition key and the Azure portal doesn't allow creation of containers without partition key. The emulator also doesn't allow it. Containers without partitioned key are deprecated and only way to create those is via Powershell, CLI or old SDK. Behind the scene, the Cosmos DB is setting the partition key as "_partitionKey". Any containers created using Portal or CLI with partition key specified exactly as "_partitionKey" fails to work with MassTransit with error message as
PartitionKey value must be supplied for this operation.
System.InvalidOperationException: PartitionKey value must be supplied for this operation.
at Microsoft.Azure.Documents.Client.DocumentClient.<AddPartitionKeyInformationAsync>d__344.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Client.DocumentClient.<ReadDocumentPrivateAsync>d__233.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MassTransit.DocumentDbIntegration.Saga.DocumentDbSagaRepository`1.<Send>d__16`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MassTransit.Saga.Pipeline.Filters.CorrelatedSagaFilter`2.<GreenPipes-IFilter<MassTransit-ConsumeContext<TMessage>>-Send>d__5.MoveNext()
This can be reproduced in one of MassTransit tests as well. The test would pass if we allow the tests to create the DB and container but it fails if we pre-create the DB and container (with the partition key set as "_partitionKey"). I have compared the Settings for the auto created container and manually created container and from what we have access to, there isn't any difference (See screenshot)
Test Name: Should_have_removed_the_state_machine
Test Outcome: Failed
Result Message: System.InvalidOperationException : PartitionKey value must be supplied for this operation.
Result StandardOutput:
17:34:14.063-A Start MassTransit.Transport.Send |e54a1ba8-452425b175a04fe0.
17:34:14.150-A Start MassTransit.Transport.Receive |e54a1ba8-452425b175a04fe0.1. |e54a1ba8-452425b175a04fe0.
17:34:14.150-D SEND loopback://localhost/input_queue 7a000000-9a3c-0005-c0b7-08d7eb0be083 MassTransit.DocumentDbIntegration.Tests.GirlfriendYelling
17:34:14.153-A Stop MassTransit.Transport.Send |e54a1ba8-452425b175a04fe0. -- 92ms
17:34:14.268-A Start MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.1. |e54a1ba8-452425b175a04fe0.1.
17:34:14.287-A Stop MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.1. |e54a1ba8-452425b175a04fe0.1. 18ms
17:34:14.292-W R-RETRY loopback://localhost/input_queue 7a000000-9a3c-0005-c0b7-08d7eb0be083
System.InvalidOperationException: PartitionKey value must be supplied for this operation.
at Microsoft.Azure.Documents.Client.DocumentClient.<AddPartitionKeyInformationAsync>d__344.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.Client.DocumentClient.<ReadDocumentPrivateAsync>d__233.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
at Microsoft.Azure.Documents.BackoffRetryUtility`1.<ExecuteRetryAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at MassTransit.DocumentDbIntegration.Saga.Context.DocumentDbDatabaseContext`1.<Load>d__16.MoveNext() in C:\Code\Samples\MassTransit-develop\src\Persistence\MassTransit.DocumentDbIntegration\Saga\Context\DocumentDbDatabaseContext.cs:line 51
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at MassTransit.DocumentDbIntegration.Saga.Context.DocumentDbSagaRepositoryContext`2.<Load>d__7.MoveNext() in C:\Code\Samples\MassTransit-develop\src\Persistence\MassTransit.DocumentDbIntegration\Saga\Context\DocumentDbSagaRepositoryContext.cs:line 63
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at MassTransit.Saga.SendSagaPipe`2.<Send>d__5.MoveNext() in C:\Code\Samples\MassTransit-develop\src\MassTransit\Saga\SendSagaPipe.cs:line 36
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at MassTransit.Saga.Pipeline.Filters.CorrelatedSagaFilter`2.<GreenPipes-IFilter<MassTransit-ConsumeContext<TMessage>>-Send>d__5.MoveNext() in C:\Code\Samples\MassTransit-develop\src\MassTransit\Saga\Pipeline\Filters\CorrelatedSagaFilter.cs:line 52
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at MassTransit.Saga.Pipeline.Filters.CorrelatedSagaFilter`2.<GreenPipes-IFilter<MassTransit-ConsumeContext<TMessage>>-Send>d__5.MoveNext() in C:\Code\Samples\MassTransit-develop\src\MassTransit\Saga\Pipeline\Filters\CorrelatedSagaFilter.cs:line 70
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at GreenPipes.Filters.RetryFilter`1.<GreenPipes-IFilter<TContext>-Send>d__4.MoveNext()
17:34:14.330-A Start MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.2. |e54a1ba8-452425b175a04fe0.1.
17:34:14.331-A Stop MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.2. |e54a1ba8-452425b175a04fe0.1. 927700ns
17:34:14.332-A Start MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.3. |e54a1ba8-452425b175a04fe0.1.
17:34:14.333-A Stop MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.3. |e54a1ba8-452425b175a04fe0.1. 739200ns
17:34:14.333-A Start MassTransit.Saga.Send |e54a1ba8-452425b175a04fe0.1.4. |e54a1ba8-452425b175a04fe0.1.
Is there a way to configure the MassTransit Saga Persistence to be able to work with containers created with partition key? The Azure Portal already doesn't allow containers without a partition key and eventually the CLI would also not allow (the commands are already deprecated).
We're seeing this error when starting up a process that uses HangFire (1.6.22) and PostgreSQL.
2019-03-19 00:12:45,406 [1] ERROR
Hangfire.PostgreSql.PostgreSqlStorage [(null)] - Error while executing install/upgrade Npgsql.PostgresException (0x80004005): 42701: column "updatecount" of relation "lock" already exists
at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Npgsql.NpgsqlDataReader.<NextResult>d__46.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlDataReader.NextResult()
at Npgsql.NpgsqlCommand.<ExecuteDbDataReader>d__100.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlCommand.<ExecuteNonQuery>d__92.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Npgsql.NpgsqlCommand.ExecuteNonQuery()
fixed. This may help others , what we did.
The real issue was , when we created Tables for HangFire it was created using SQL scripts that available in source repo. And when code run it found some mismatch and automatically try to update DB tables, and failed.
So we deleted all tables and let HangFire create tables on first run. Now working fine.
When performing a failover on our cluster, the following EndOfStreamException is thrown every time (but only once) before the application successfully utilises the newly elected primary.
I am using 2.5.0 of the .NET Driver and also providing the replica set name in the connection string:
var url = MongoUrl.Create("mongodb://<dbuser>:<dbpassword>#ds012345-a0.mlab.com:56789,ds012345-a1.mlab.com:56790/<dbname>?replicaSet=rs-ds012345");
var client = new MongoClient(new MongoClientSettings()
{
Servers = url.Servers,
Credentials = credentials
});
Exception:
MongoDB.Driver.MongoConnectionException: An exception occurred while
receiving a message from the server. --->
System.IO.EndOfStreamException: Attempted to read past the end of the
stream. at
MongoDB.Driver.Core.Misc.StreamExtensionMethods.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Connections.BinaryConnection.d__51.MoveNext()
--- End of inner exception stack trace --- at MongoDB.Driver.Core.Connections.BinaryConnection.d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Connections.BinaryConnection.d__52.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Connections.BinaryConnection.d__54.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.WireProtocol.CommandWireProtocol1.<ExecuteAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Servers.Server.ServerChannel.<ExecuteProtocolAsync>d__261.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Operations.CommandOperationBase1.<ExecuteProtocolAsync>d__29.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Operations.ReadCommandOperation1.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Operations.FindCommandOperation1.<ExecuteAsync>d__111.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.Core.Operations.FindOperation1.d__111.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.OperationExecutor.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.MongoCollectionImpl1.d__601.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
MongoDB.Driver.IAsyncCursorSourceExtensions.<ToListAsync>d__161.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
I was also facing the same error, but it was coming alternatively. Once request success and second fail with the above error.
To solve this issue, I used using like below
using (var cursor = await _mongoDb.GetCollection<BsonDocument>(CollectionName)
.FindAsync(queryJson.ToString(), options))
{
//operation
}
Please check if this resolves your issues too.
Thanks
We recently moved our API:s to Azure App Service and started getting some random (connection?) timeouts to our database:
System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. ---> Npgsql.NpgsqlException: Exception while reading from stream ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.ReadBuffer.<Ensure>d__27.MoveNext()
--- End of inner exception stack trace ---
at Npgsql.ReadBuffer.<Ensure>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<DoReadMessage>d__148.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Npgsql.NpgsqlConnector.<ReadPrependedMessages>d__150.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<ReadMessage>d__147.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Npgsql.NpgsqlConnector.<ReadExpecting>d__154`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Npgsql.NpgsqlDataReader.<NextResult>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlDataReader.<<NextResultAsync>b__31_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlCommand.<Execute>d__71.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Npgsql.NpgsqlCommand.<ExecuteDbDataReader>d__92.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Npgsql.NpgsqlCommand.<>c__DisplayClass90_0.<<ExecuteDbDataReaderAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.<ExecuteAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable`1.AsyncEnumerator.<BufferlessMoveNext>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Storage.Internal.NpgsqlExecutionStrategy.<ExecuteAsync>d__7`2.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Storage.Internal.NpgsqlExecutionStrategy.<ExecuteAsync>d__7`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable`1.AsyncEnumerator.<MoveNext>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Linq.AsyncEnumerable.<FirstOrDefault_>d__165`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.TaskResultAsyncEnumerable`1.Enumerator.<MoveNext>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.<MoveNextCore>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Linq.AsyncEnumerable.AsyncIterator`1.<MoveNext>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.<MoveNext>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<ExecuteSingletonAsyncQuery>d__23`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at OpenIddict.OpenIddictProvider`4.<HandleTokenRequest>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at AspNet.Security.OpenIdConnect.Server.OpenIdConnectServerHandler.<InvokeTokenEndpointAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at AspNet.Security.OpenIdConnect.Server.OpenIdConnectServerHandler.<HandleRequestAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.<Invoke>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()
This specific exception is thrown from OpenIddict which we use for authentication, but we've seen similar being thrown from our own code as well.
I haven't been able to find any common denominator, these timeouts occur randomly and not under very heavy load (not that many simultaneous requests). In our logs I can see that the exception is thrown ~20 seconds after the requests comes in.
Our API is a .NET Core application deployed to Azure App Service (Basic B3).
The database is Amazon Aurora (PostgreSQL).
I have made sure that the Azure outbound IPs are white listed in the Amazon security group for the database. I have event tried to disable the database firewall (because I believed Azure was using another IP), but still received the same error.
Neither the database or the API is under heavy load when these problems occur.
I'm not sure how to debug this any further? What can I do to find and fix the root cause? Any help is appreciated.
Check your connection string, are you connecting to the cluster node or the instance? Connecting to the instance can cause these errors occasionally, connecting to the cluster node shields you from this.
2017-07-02 17:37:51.8708|INFO|thread: 5|AsyncReaderWriterLock|Released concurrent scheduler for operation: 10 at GitClient.cs:290 after blocking 0.4168211s
2017-07-02 17:37:52.2268|INFO|thread: 1|ObservableEx|Could not refresh repository information from API failed.
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Octokit.Internal.HttpClientAdapter.<Send>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Octokit.Connection.<RunRequest>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Octokit.Connection.<Run>d__23`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Octokit.ApiConnection.<Get>d__0`1.MoveNext()
2017-07-02 17:37:52.2318|INFO|thread: 1|AsyncReaderWriterLock|Acquiring concurrent scheduler for operation 12 at BranchesViewModel.cs:356
2017-07-02 17:37:52.2318|INFO|thread: 4|AsyncReaderWriterLock|Acquired concurrent scheduler for operation: 12 at BranchesViewModel.cs:356 after waiting 6.27E-05s
2017-07-02 17:37:52.2318|INFO|thread: 1|AsyncReaderWriterLock|Acquiring concurrent scheduler for operation 13 at RepositoryViewModel.cs:318
2017-07-02 17:37:52.2318|INFO|thread:14|AsyncReaderWriterLock|Acquired concurrent scheduler for operation: 13 at RepositoryViewModel.cs:318 after waiting 6.9E-05s
2017-07-02 17:37:52.2548|INFO|thread: 5|AsyncReaderWriterLock|Released concurrent scheduler for operation: 12 at BranchesViewModel.cs:356 after blocking 0.0226196s
2017-07-02 17:37:52.2758|INFO|thread: 1|NewPullRequestViewModel|No matching pull request found, clearing selection...
What operation did you make before you get this Error?
I guess something goes wrong with your config file.
Check your username,password,port,and so on.(use git config --list)