AppFabricCaching services are crashing with the below msg logged:
AppFabric Caching service crashed.{Microsoft.ApplicationServer.Caching.ConfigStoreException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) ---> System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) ---> System.ComponentModel.Win32Exception: The specified network name is no longer available
--- End of inner exception stack trace ---
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Microsoft.ApplicationServer.Caching.SqlServerCustomProviderCore.GetEntries(Object transactionContext, String type)
--- End of inner exception stack trace ---
at Microsoft.ApplicationServer.Caching.SqlServerCustomProviderCore.GetEntries(Object transactionContext, String type)
at Microsoft.ApplicationServer.Caching.SqlServerCustomProvider.<>c__DisplayClass1`1.<PerformOperation>b__0()
at Microsoft.ApplicationServer.Caching.SqlServerCustomProvider.PerformOperation(Action action)
at Microsoft.ApplicationServer.Caching.SqlServerCustomProvider.PerformOperation[TResult](Func`1 func)
at Microsoft.ApplicationServer.Caching.ClusterConfigDictionaryReader.GetConfigs[T](String type)
at Microsoft.ApplicationServer.Caching.ServiceConfigurationManager.InitializeHostsData(IClusterConfigurationReader globalConfig)
at Microsoft.ApplicationServer.Caching.ServiceConfigurationManager.InitializeDataFromGlobalConfig()
at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartService(Boolean deleteTKT)
at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(Object context)}
any pointers are very much appreciated.
This message is very important
A transport-level error has occurred when receiving results from the
server. (provider: TCP Provider, error: 0 - The specified network name
is no longer available.) ---> System.Data.SqlClient.SqlException: A
transport-level error has occurred when receiving results from the
server. (provider: TCP Provider, error: 0 - The specified network name
is no longer available.) ---> System.ComponentModel.Win32Exception:
The specified network name is no longer available
This error suggests that the Sql Server instance hosting the cluster configuration is no longer available. As the configuration location can be a single point of failure, it have to be always available.
Check the configured connection string in DistributedCacheService.exe.config under Program Files\ Microsoft AppFabric 1.1 for Windows Server.
Related
I have a problem with a normal SQL database restored from a DB set in AlwaysOn high availability group in SQL Server 2017.
I restored a copy of production db to a different server, to be used as QA test database, with a different name also - MyDB_demo
The problem is, the QA app copy (same code as production with new development enhancements) get an error at some point.
Even if my conn str points to MyDB_demo, I get the following error
[SqlException (0x80131904): The target database ('MyDB') is in an availability group and is currently accessible for connections when the application intent is set to read only. For more information about application intent, see SQL Server Books Online.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action 1 wrapCloseInAction) +2444190
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action 1 wrapCloseInAction) +5775712
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +285
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4169
System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +58
System.Data.SqlClient.SqlDataReader.get_MetaData() +89
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption) +409
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) +2127
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) +911
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +64
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +240
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior) +41
System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior) +12
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +139
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +136
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +88
MyApp.SqlHelper.ExecuteDataset(SqlConnection connection, CommandType commandType, String commandText, SqlParameter[] commandParameters) +163
MyApp.PermitFunctions.GetSystemMessages(String sp, Int32 iPermitID, Int32 iAppID, SqlConnection cn) +219
MyApp.Municipality.LoadSystemMessage() +3869
MyApp.Municipality.Page_Load(Object sender, EventArgs e) +101
System.Web.UI.Control.OnLoad(EventArgs e) +95
System.Web.UI.Control.LoadRecursive() +59
System.Web.UI.Control.LoadRecursive() +131
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +678
Is there any reference in the newly restored DB (named now MyDB_demo) that stores the original name of production DB and why is it trying to access it?
Any suggestion is appreciated.
EDIT
Actually, the server used to restore MyDB_demo is one of the secondary nodes for AlwasyOn availability group; it also contains a RO copy of production database, MyDB.
So the server has:
RO copy of production DB (MyDB)
normal, stand-alone db restored for QA - MyDB_demo
Hence, I understand the error message - it would makes sense if I tried to access directly the secondary, RO copy of production db from connection string.
But I do not: the connection string (which I double-checked) is trying to connect to QA db, MyDB_demo.
Here is some additional info:
the error is thrown in SQLHelper class, the helper class from MS to work with SQL Server, in the ExecuteDataset function
the error is thrown ONLY on one stored procedure - lot of other stored procedures and also direct SQL statements run just fine
I inspected the stored procedure, thinking it might contains accidentally a hardcoded reference to DB name - it doesn't
and the strange part - I run the stored procedure with the same parameters as called from the app in SSMS - and it run just fine - no error
So it looks somehow the connection string MIGHT be altered (!!!) is some way by the NET application itself, and only for this stored procedure?
Anyone ever encountered something like this?
Thank you
Due to the strange (read: stupid) situation that the culprit SP fails only when called from within app, but runs ok when tried in SSMS, I tried a "stupid" approach: I inspected its code, commented out two fields that were set with sub-selects like select top 1 from ..... where.... (actually I replaced theit values with dummy ones) and I changed an Order By field that was initially specified like "InspectionType" Desc, which I removed quotation marks from.
Doing this, the SP suddenly started to work ok even when called from the app.
Then I reverted all changes to original (added quoted back and put back the sub-selects) and the SP continued to work ok.
So ... problem solved.
Stupid approach for stupid problem (!?!?!)
In any case, if anyone has a better idea or explanation of what might have happened, I'd be glad to hear it
EDIT
I think I understand the fix.
By editing and saving the stored procedure, its query plan was recompiled.
So the original error might have been caused by the old query plan.
But why did it referenced the database name? Is the actual database name referenced in query plans? This looks a little odd to me.
And another question (open):
Does the SQL Server optimizer detect if a DB runs in high-availability mode, and when optimizing the queries, does it decide if a query is read-only mode and automatically redirects it toward a read-only node? Even if the ApplicationIntent readonly parameter is not present in connection string?
Because it was not in this case, even in production - we just implemented AlwaysOn functionality and are in the process of updating app to take advantage of R/O nodes.
Any comments are appreciated
I have a shared hosting site "production" that works perfectly. It has users and thus far I have no issues.
I decided to created a test environment. I created another Azure app and copied the code from my "production" site to this test site. This happen today. It has no users. I created a DB and I agreed to pay five bucks a month. So this is not a free account. I uploaded my seed data which part of it was 40K rows of zip code. That went fine without issues.
However, when I use the site, I consistently get timeout error. I can assure you that this is NOT because of usage. I increased the timeout from 30 to 90 in the connection string but that didn't make a difference. There is no code difference between the production and test.
I have read many post about this but none helped. Here are some samples:
post1 post2 post3 post4 post5 post6
Any ideas?
> [Win32Exception (0x80004005): The wait operation timed out]
>
> [SqlException (0x80131904): Timeout expired. The timeout period
> elapsed prior to completion of the operation or the server is not
> responding. This failure occurred while attempting to connect to the
> routing destination. The duration spent while attempting to connect to
> the original server was - [Pre-Login] initialization=27; handshake=25;
> [Login] initialization=0; authentication=0; [Post-Login] complete=3;
> ] System.Data.SqlClient.SqlConnection.OnError(SqlException
> exception, Boolean breakConnection, Action1 wrapCloseInAction)
> +2418094 System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
> exception, Boolean breakConnection, Action1 wrapCloseInAction)
> +5694436 System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
> stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +285
> System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior,
> SqlCommand cmdHandler, SqlDataReader dataStream,
> BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
> stateObj, Boolean& dataReady) +3731
> System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +58
> System.Data.SqlClient.SqlDataReader.get_MetaData() +89
> System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds,
> RunBehavior runBehavior, String resetOptionsString) +379
> System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior
> cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean
> async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader
> ds, Boolean describeParameterEncryptionRequest) +2064
> System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
> cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
> method, TaskCompletionSource`1 completion, Int32 timeout, Task& task,
> Boolean asyncWrite) +375
> System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
> cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
> method) +53
> System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
> behavior, String method) +240
> System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior
> behavior) +41
> System.Data.Common.DbCommand.ExecuteReader(CommandBehavior behavior)
> +12 System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand
> entityCommand, CommandBehavior behavior) +435
It was the DTU issue. I had 5 DTU (S0) now I have 20 DTU (S1). If I have my physics right it's like a steam engine and BTU. Horsepower matters.
I was getting the same error but after looking a little deeper the issue I found was different and would like to share it if it helps someone else. I had an EntityFramework migration that ran fine on local development server and staging server but was taking too long on production. It was causing the timeout and since it was not a migration failed exception we did not think that would be the problem.
In this migration we were indexing a table that had millions of records hence indexing was taking longer and EF runs migration first time a connection is requested once deployed was causing issue.
Since my table had data that could be regenerated (nothing critical to keep just some summary data) I quickly empties the table and ran migrations. Everything was good after that.
We've got the following DB Configuration setup, however we're still seeing a large volume of transient error notifications when operating against SQL Azure (P1 tier).
public class DbConfiguration : System.Data.Entity.DbConfiguration
{ public DbConfiguration()
{
SetExecutionStrategy("System.Data.SqlClient", () => new SqlAzureExecutionStrategy(5, TimeSpan.Parse("00:00:03"));
}
}
Can anyone familiar with this please comment on anything we may be setting up incorrectly? We are using EF6.
The exact exception we are getting is:
An exception has been raised that is likely due to a transient failure. If you are connecting to a SQL Azure database consider using SqlAzureExecutionStrategy.
Exception Full Value: System.Data.Entity.Core.EntityException: An exception has been raised that is likely due to a transient failure. If you are connecting to a SQL Azure database consider using SqlAzureExecutionStrategy. ---> System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> System.Data.Entity.Core.EntityException: An exception has been raised that is likely due to a transient failure. If you are connecting to a SQL Azure database consider using SqlAzureExecutionStrategy. ---> System.Data.SqlClient.SqlException: A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception: An existing connection was forcibly closed by the remote host
--- End of inner exception stack trace ---
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at System.Data.Entity.Infrastructure.Interception.InternalDispatcher1.Dispatch[TTarget,TInterceptionContext](TTarget target, Action2 operation, TInterceptionContext interceptionContext, Action3 executing, Action3 executed)
at System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(DbConnection connection, DbInterceptionContext interceptionContext)
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.<>c__DisplayClass1.b__0()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation)
--- End of inner exception stack trace ---
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation)
at System.Data.Entity.Core.EntityClient.EntityConnection.Open()
--- End of inner exception stack trace ---
at System.Data.Entity.Core.EntityClient.EntityConnection.Open()
at System.Data.Entity.Core.Objects.ObjectContext.EnsureConnection(Boolean shouldMonitorTransactions)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
at System.Data.Entity.Core.Objects.ObjectContext.<>c__DisplayClass471.b__45()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation)
--- End of inner exception stack trace ---
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func1 operation)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteFunction[TElement](String functionName, ExecutionOptions executionOptions, ObjectParameter[] parameters)
Transient errors occur in Azure due to below reasons(not an elaborate list..)
1.Your database is under heavy load(using up all/reaching dtu quota)
Use below DMV to see DTU metrics
SELECT
(COUNT(end_time) - SUM(CASE WHEN avg_cpu_percent > 80 THEN 1 ELSE 0 END) * 1.0) / COUNT(end_time) AS 'CPU Fit Percent'
,(COUNT(end_time) - SUM(CASE WHEN avg_log_write_percent > 80 THEN 1 ELSE 0 END) * 1.0) / COUNT(end_time) AS 'Log Write Fit Percent'
,(COUNT(end_time) - SUM(CASE WHEN avg_data_io_percent > 80 THEN 1 ELSE 0 END) * 1.0) / COUNT(end_time) AS 'Physical Data Read Fit Percent'
FROM sys.dm_db_resource_stats
2.If you see DTU usage is normal at the time of transient failures,but still seeing errors,it would be good to see if there is a chance of optimization.We used to have same failures,we optimized sql code and did index rebuild and stats
3.reach out to Azure Support with the timestamps of failures to see if there is any maintenance activity
It looks like you are using the ‘SqlAzureExecutionStrategy’ in the entity framework which in turn calls the ‘DbExecutionStrategy’ which has exponential backoff as the inbuilt mechanism. The can lead to greater delays I would recommend the following:
1) Retrying linearly after 5.5sec is ideal. The client caches the error message for 5sec so even if you retry within this time period it will just send you the cached message.
2) The timeout recommended is 30 sec.
3) Also I would highly recommend connection pooling, the details are here:
https://msdn.microsoft.com/library/8xx3tyca.aspx
Also can you please send me the server name and the database name at dmalik#Microsoft.com if possible with the duration of the outages and frequencies?
After research I found that it was related to the limits of Azure SQL Database maximum logins. I was using 'Basic' service tire and max concurrent users can login with that is 30.
Azure has pricing tiers that have quite dramatic differences in performance. To achieve that, they throttle a lot of performance metrics, e.g. CPU power, requests per minute, etc.
This means that if you're pushing over your tier, your requests will start getting queued up as the CPU power / volume of requests is too high to process. This results in timeouts and then the request limit grows as requests wait to be processed. Eventually, it gets to the point where the database essentially goes down.
My experience is that the lower database levels, such as S0 and S1, are under-powered and shouldn't be used for anything other than development or very basic sites.
There are some great tools in the Azure portal that allow you to debug what is going on with your database, such as the CPU graphs, index advisor and query performance insights.
*Here are the related links: *
https://blogs.technet.microsoft.com/latam/2015/06/01/how-to-deal-with-the-limits-of-azure-sql-database-maximum-logins/
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-dtu-resource-limits-single-databases
You need to upgrade the subscription in the Azure if that solve your problem. or Enable an execution strategy as mentioned here: https://msdn.microsoft.com/en-us/data/dn456835.aspx .
Hope this helps !!
Thanks.
I am synching my production SQL Server 2008 R2 db with my failover SQL Server 2008 R2 Express server using Microsoft Sync Framework 2.1 and it worked great for a while and then after a few days of syncs the source (production SQL Server) starts running out of memory (RAM) because of the syncs.
It didn't stop running or anything or cause website transactions to fail but the Sync Framework starts throwing the below error if there is low ram on production box.
Anyone seen this or have any idea why it is doing this and if there is a fix?
Here is the error ..
Microsoft.Synchronization.Data.DbSyncException: An unexpected exception occurred in a background thread that is enumerating
changes. See the inner exception for more details.
System.Data.SqlClient.SqlException: There is insufficient system
memory in resource pool 'internal' to run this query.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds,
RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean
async, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method, TaskCompletionSource1 completion, Int32 timeout, Task& task,
Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at System.Data.SqlClient.SqlCommand.DeriveParameters()
at System.Data.SqlClient.SqlCommandBuilder.DeriveParameters(SqlCommand
command)
at Microsoft.Synchronization.Data.ManagementUtils.DeriveParameters(SqlCommand
command, IEnumerable`1 columns)
at Microsoft.Synchronization.Data.SqlServer.SqlSyncAdapter.SetCommandSessionParameters(IDbCommand
command, String commandName)
at Microsoft.Synchronization.Data.RelationalSyncProvider.EnumerateChangesInBatchesInternal(Object
batchProducer)
--- End of inner exception stack trace ---
at Microsoft.Synchronization.Data.DbSyncBatchProducer.DequeueBatch()
at Microsoft.Synchronization.Data.RelationalSyncProvider.ConsumeBatchFromProducer(DbSyncScopeMetadata
scopeMetadata)
at Microsoft.Synchronization.Data.RelationalSyncProvider.GetChanges(DbSyncScopeMetadata
scopeMetadata, DbSyncSession DbSyncSession, UInt32 memoryBatchSize)
at Microsoft.Synchronization.Data.RelationalSyncProvider.GetChangeBatch(UInt32
batchSize, SyncKnowledge destinationKnowledge, Object&
changeDataRetriever)
at Microsoft.Synchronization.KnowledgeProviderProxy.GetChangeBatch(UInt32
dwBatchSize, ISyncKnowledge pSyncKnowledge, ISyncChangeBatch&
ppChangeBatch, Object& ppUnkDataRetriever)
at Microsoft.Synchronization.CoreInterop.ISyncSession.Start(CONFLICT_RESOLUTION_POLICY
resolutionPolicy, _SYNC_SESSION_STATISTICS& pSyncSessionStatistics)
at Microsoft.Synchronization.KnowledgeSyncOrchestrator.DoOneWaySyncHelper(SyncIdFormatGroup
sourceIdFormats, SyncIdFormatGroup destinationIdFormats,
KnowledgeSyncProviderConfiguration destinationConfiguration,
SyncCallbacks DestinationCallbacks, ISyncProvider sourceProxy,
ISyncProvider destinationProxy, ChangeDataAdapter
callbackChangeDataAdapter, SyncDataConverter conflictDataConverter,
Int32& changesApplied, Int32& changesFailed)
at Microsoft.Synchronization.KnowledgeSyncOrchestrator.DoOneWayKnowledgeSync(SyncDataConverter
sourceConverter, SyncDataConverter destinationConverter, SyncProvider
sourceProvider, SyncProvider destinationProvider, Int32&
changesApplied, Int32& changesFailed)
at Microsoft.Synchronization.KnowledgeSyncOrchestrator.Synchronize()
at Microsoft.Synchronization.SyncOrchestrator.Synchronize()
try enabling batching by setting the MemoryDataCacheSize property.
I try to discover service endpoint .../XRMServices/2011/Organization.svc and I've this error:
An error has occurred.
Try this action again. If the problem continues, check the Microsoft Dynamics CRM Community for solutions or contact your organization's Microsoft Dynamics CRM Administrator. Finally, you can contact Microsoft Support.
System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Crm.MainApplication.Application_Error(Object sender, EventArgs eventArguments)
Stack trace:
Error Description:
Generic SQL error.
Error Details:
Generic SQL error.
Full Stack:
[SqlException: Cannot open database "ASPState" requested by the login. The login failed.
Login failed for user 'NT AUTHORITY\NETWORK SERVICE'.]
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, TimeoutTimer timeout)
at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(SqlConnection owningObject, TimeoutTimer timeout, SqlConnectionString connectionOptions, String newPassword, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, String newPassword, SqlConnection owningObject, Boolean redirectedUserInstance)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnection owningConnection, DbConnectionPool pool, DbConnectionOptions options)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at System.Web.SessionState.SqlSessionStateStore.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo, TimeSpan retryInterval)
[HttpException: Unable to connect to SQL Server session database.]
at System.Web.SessionState.SqlSessionStateStore.ThrowSqlConnectionException(SqlConnection conn, Exception e)
at System.Web.SessionState.SqlSessionStateStore.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo, TimeSpan retryInterval)
at System.Web.SessionState.SqlSessionStateStore.GetConnection(String id, Boolean& usePooling)
at System.Web.SessionState.SqlSessionStateStore.DoGet(HttpContext context, String id, Boolean getExclusive, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
at System.Web.SessionState.SqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
What may cause the problem, what is wrong with CRM?
Since you are using NETWORK SERVICE, you need to make sure the webserver's computer account is added to the SQLAccessGroup in Active Directory. Normally this is done for you during CRM setup, but if you want to change the account later you must remember to add the new account to this group manually. Here is a link that describes the different groups that CRM uses:
http://ayazahmad.wordpress.com/2007/01/26/active-directory-security-groups-for-mscrm/
I've seen a similar message when the user did not have a role assigned, but that occured in a test enviroment, still worth checking.
I completely re-installed development stack and the same happened again, I re-installed my virtual machine and it worked, seems like it some unknown issue. Now I'll try to reproduce the same errors on a new instances of a virtual machine.
Thanks to all)
I have seen also this message on my CRM server. What I've done is to just enable and disable anonymous authentication on IIS, and thing worked for me. Also I've played with web.config, service settings, but later I get that section back as it was on first install of CRM.
In my oppinion, IIS doesn't tell WCF how to auth, and couple of cold restarts, and enabling and disabling anonymous auth. wake up WCF.
Your AppPool needs to be a User that has access to the database.
With regards to the WCF Service,
Check your IIS Website Authentication settings, for WCF Services you need
Anonymous Authentication Enabled
ASP.NET Impersonation Enabled
Windows Authentication Enabled
for On-Premise installation running in an AD environment.
We also had the issue and it was resolved after restarting IIS.