Azure deployment is cycling for a long time - deployment

12:08:41 - Preparing deployment for Windows Azure MSDN - Visual Studio Ultimate with
Subscription ID: xxxxxxxxxxxxxxxxxxxx...
12:08:41 - Connecting...
12:08:41 - Verifying storage account 'cck'...
12:08:43 - Uploading Package...
12:41:00 - Creating...
12:57:14 - Created Deployment ID: xxxxxxxxxxxxxxxxx.
12:57:14 - Starting...
12:57:55 - Initializing...
12:57:55 - Instance 0 of role RIS2048.ConsultaClick.Web is in an unknown state
12:59:03 - Instance 0 of role RIS2048.ConsultaClick.Web is starting the virtual machine
13:00:40 - Instance 0 of role RIS2048.ConsultaClick.Web is in an unknown state
13:01:48 - Instance 0 of role RIS2048.ConsultaClick.Web is busy
13:06:12 - Instance 0 of role RIS2048.ConsultaClick.Web is cycling
It's now 13:31, more than 1 hour stared ago and at 25 minutes the instance is cycling (I don't know what that means). Will it finish? When?
Here's part of my ServiceDefinition.csdef file with 2 VirtualApplications:
<Site name="Web">
<VirtualApplication name="CCKPt"
physicalDirectory=".">
<VirtualDirectory name="images"
physicalDirectory="..\RIS2048.ConsultaClick.WWWPacientes\imgpt" />
</VirtualApplication>
<VirtualApplication name="CCKRo"
physicalDirectory=".">
<VirtualDirectory name="images"
physicalDirectory="..\RIS2048.ConsultaClick.WWWPacientes\imgro" />
</VirtualApplication>
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint1" />
</Bindings>
</Site>

Is this a new app you just built? And does it run locally in the emulator? First thing I always check is the connection string for diagnostics, along with the session state provider. By default, your diagnostics are written to the emulator storage (meaning SQL Express on your local machine), and SQL Express is used for session state, which doesn't exist in Windows Azure. Rather, you'd need to switch it to Cache or SQL Azure.

Likely means there's a problem. Sometimes, if you're lucky, you can squeeze a little more info about where exactly it's bombing from IntelliTrace:
http://blogs.msdn.com/b/jnak/archive/2010/06/07/using-intellitrace-to-debug-windows-azure-cloud-services.aspx

Related

Publishing an azure web service takes 30 min or more? Is this normal?

The azure powershell command Publish-AzureServiceProject is taking upto 30min or more to successfully publish an application. We run the following azure powershell command.
Publish-AzureServiceProject
-Package
'C:\tca\tick\712fd6c47ab9a6e5\azurewebsite\Weingartner.Distribution.Azure.cspkg'
-Slot Staging
-Configuration
'C:\tca\tick\712fd6c47ab9a6e5\azurewebsite\ServiceConfiguration.Cloud.cscfg'
which results in the following output
Id Type Subscriptions Tenants
-- ---- ------------- -------
research#weingartnergmbh.onmicrosoft.com User XXXXXXX {XXXXXXXXXXX...
14:28:50 - Verifying storage account 'weingartnerdistribution'...
Publishing weingartnerdistribution to Microsoft Azure. This may take several minutes...
14:28:51 - Connecting...
14:28:53 - Uploading Package to storage service weingartnerdistribution...
14:29:32 - Starting...
14:30:37 - Created Deployment ID: XXXXXXXXXXXXXXXXXXXXXXX.
14:30:37 - Initializing...
15:02:15 - Instance
Weingartner.Distribution.Website.Hosting.Azure_IN_0
of role
Weingartner.Distribution.Website.Hosting.Azure
is busy.
15:06:24 - Instance
Weingartner.Distribution.Website.Hosting.Azure_IN_0
of role
Weingartner.Distribution.Website.Hosting.Azure
is ready.
15:06:25 - Created Website URL: http://XXXXXXXXXX.cloudapp.net/.
Finished Target: Publish
It takes 30 minutes to initialize the VM. When I go to the azure portal there are no errors that are obvious and the service runs correctly when eventually it starts.
I would like to ask azure what is going on but as I'm using an extra small instance and have no service contract ie:
Question? Do people with higher paid plans or larger spec virtual machines see the same 30 min turn around for publishing an upgrade to an app service or is this is purely technical problem that can be solved with better usage of the APIs?
EDIT
I ran the following command at the command line at 7:28am
Publish-AzureServiceProject -Package .\Weingartner.Distribution.Azure.cspkg -Configuration .\ServiceConfiguration.Cloud.cscfg -Slot Staging
and according to the AzurePortal it finished at 7:38am
However the debug output from the poweshell script was continually outputting the following until 7:53am GMT+1 ( Sensitive data is erased with XXXXX and note the logs are in GMT so add one hour )
Headers:
x-ms-servedbyregion : ussouth3
Strict-Transport-Security : max-age=31536000; includeSubDomains
x-ms-request-id : XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Cache-Control : no-cache
Date : Tue, 09 Jan 2018 06:53:31 GMT
Server : 1.0.6198.592,(rd_rdfe_stable.180102-1534),Microsoft-HTTPAPI/2.0
Body:
<Deployment xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<Name>s2018-01-09-07-25-41-2872</Name>
<DeploymentSlot>Staging</DeploymentSlot>
<PrivateID>XXXXXXXXXXXXXXXX</PrivateID>
<Status>Running</Status>
<Label>XXXXXXXXX</Label>
<Url>http://XXXXXXXXXXXXXX.cloudapp.net/</Url>

mongo-csharp-driver 2.0 / nservicebus.mongodb 2.1 can no longer connect to Mongo

We recently updated our NserviceBus.MongoDb package from 2.0.3 to 2.1. This also updated the mongo-csharp-driver package from 1.10 to 2.0.1. We now see the following exception when the service starts:
2015-07-08 11:29:16,589 [1] WARN NServiceBus.MongoDB.Internals.MongoHelpers [(null)] <(null)> - Mongo could not be contacted using: server201.localco.test:27017.
If you are using a Replica Set, please ensure that all the Mongo instance(s) server201.localco.test:27017 are available.
To configure NServiceBus to use a different connection string add a connection string named "NServiceBus/Persistence" in your config file.
Reason: System.InvalidOperationException: There is no current primary.
at MongoDB.Driver.MongoServer.Ping()
at NServiceBus.MongoDB.Internals.MongoHelpers.VerifyConnectionToMongoServer(MongoClientAccessor mongoClientAccessor) in d:\Development\NServiceBus.MongoDB\src\NServiceBus.MongoDB\Internals\MongoHelpers.cs:line 50
The only relevant line in our nservicebus endpoint config is:
configuration.UsePersistence<MongoDBPersistence>();
and our connection string is:
<add name="NServiceBus.Persistence" connectionString="mongodb://server201.localco.test/?replicaSet=rs0" />
If I uninstall/rollback 2.1/2.0.1 to 2.0.3/1.10 (nservicebus.mongodb/mongo-csharp-driver) the error goes away.
Update: I see that the database itself (with a TimoutData collection) is actually created in Mongo...
Change your connection string from
<add name="NServiceBus.Persistence" connectionString="mongodb://server201.localco.test/?replicaSet=rs0" />
to
<add name="NServiceBus/Persistence/MongoDB" connectionString="mongodb://server201.localco.test/?replicaSet=rs0" />
or leave it like above and set the name to
config
.UsePersistence<MongoDbPersistence>()
.SetConnectionStringName("NServiceBus.Persistence");
Latest version of the package from Github/Nuget has addressed this issue. See bug report

NServiceBus Worker checking in with capacity of 0

I am using NServiceBus 3.3. I am trying to get a new Pre-Prod-Environment setup.
It all works fine in production and in one of my existing Pre-Prod-Environments with my existing configurations.
But in my new Environment, I am getting my workers checking-in with a capacity of 0. (They check-in with a capacity of 1 in the working Environments).
Again, the configs are the same between the Environments. (Except for machine names of course.)
Any idea why this could happen?
This is the output of my log (with Queue names and machine names changed):
2015-05-07 10:53:33,904 [1] INFO NServiceBus.Host [(null)] - Going to activate profile: NServiceBus.Distributor, NServiceBus.Host, Version=3.3.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c
2015-05-07 10:53:33,904 [1] INFO NServiceBus.Host [(null)] - Going to activate profile: NServiceBus.Production, NServiceBus.Host, Version=3.3.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c
2015-05-07 10:53:33,919 [1] INFO NServiceBus.Host [(null)] - Going to activate profile: NServiceBus.PerformanceCounters, NServiceBus.Host, Version=3.3.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c
2015-05-07 10:53:33,935 [1] WARN Distributor.myFromQueue [(null)] - No transport configuration found so the distributor will default to one thread, for production scenarios you would want to adjust this setting
2015-05-07 10:53:33,950 [1] INFO Distributor.myFromQueue [(null)] - Endpoint configured to host the distributor, applicative input queue re routed to myFromQueue.worker#DistributorHost
2015-05-07 11:10:05,015 [Worker.13] INFO Distributor.myFromQueue [(null)] - Worker myFromQueue#WorkerMachine has started up, clearing previous reported capacity
2015-05-07 11:10:05,030 [Worker.13] INFO Distributor.myFromQueue [(null)] - Worker myFromQueue#WorkerMachine checked in with available capacity: 0
And this is the relevant part of my worker config file:
<MsmqTransportConfig NumberOfWorkerThreads="4" MaxRetries="5" />
<MessageForwardingInCaseOfFaultConfig ErrorQueue="error" />
<MasterNodeConfig Node="DistributorHost" />
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="Bus.MyMessageAssembly" Endpoint="QueueForTheDistributor#DistributorHost" />
</MessageEndpointMappings>
</UnicastBusConfig>
I had the same problem and was able to resolve it by deleting the queues (myfromqueue and myfromqueue.retries) on the worker agent. NServiceBus automatically recreated the queues and everything started processing again for me.

EF Code First Migration throws StackOverflowException on Azure Web Role

The issue occurs when executing EF 6.1.2 code first migrations in an Azure Web Role (WS 2012 R2). The same migrations run fine locally, even if I point the connection string to the (Azure) Sql Database.
The StackOverflowException is thrown by Entity Framework code, and the first line of ANY of my migrations doesn´t even get hit.
I have tried running the migrations in three different ways:
DbMigrator migrator = new DbMigrator(configuration);
migrator.Update(); // Here the exception is thrown
second:
DbMigrator migrator = new DbMigrator(configuration);
pendingMigrations = migrator.GetPendingMigrations().ToList();
foreach (string pendingMigration in pendingMigrations)
{
migrator.Update(pendingMigration); // Here the exception is thrown
}
and using web.config:
<contexts>
<context type="Superb.WorkNextDoor.EFRepository.Context.WndDbContext, Superb.WorkNextDoor.EFRepository, Version=1.0.0.0, Culture=neutral">
<databaseInitializer type="System.Data.Entity.MigrateDatabaseToLatestVersion`2[[Superb.WorkNextDoor.EFRepository.Context.WndDbContext, Superb.WorkNextDoor.EFRepository], [Superb.WorkNextDoor.EFRepository.Migrations.Migrations.WndDbMigrationsConfiguration, Superb.WorkNextDoor.EFRepository.Migrations]], EntityFramework">
</databaseInitializer>
</context>
</contexts>
I see the StackOverflowException in Visual Studio when I´m remote debugging the web role. An error is recorded in windows event log of the web role (log info at the bottom of this post).
I have two migrations. I have tried running the first one from PMC and the second from the web role but no luck.
I have tried installing .Net 4.5.2 on the server, changing the VM size from XS to S and downgrading EF to version 6.1.1. Nothing worked. Also, I tried downloading the folder "E:\sites\0" from the web role to my local computer, installed the app on IIS and attached my VS Debugger and it doesn't throw that exception. There has got to be something different between my Windows 8.1 and that Windows Server 2012 R2.
I have spent a lot of time trying different things but I don´t want to give up on this and have to execute my migrations from package manager console.
Log Name: Application Source: Application Error Date:
1/11/2015 3:21:42 AM Event ID: 1000 Task Category: (100) Level:
Error Keywords: Classic User: N/A Computer:
RD0003FF508F5B Description: Faulting application name: w3wp.exe,
version: 8.5.9600.16384, time stamp: 0x5215df96 Faulting module name:
clr.dll, version: 4.0.30319.34014, time stamp: 0x52e0b86c Exception
code: 0xc00000fd Fault offset: 0x0000000000195499 Faulting process id:
0xc60 Faulting application start time: 0x01d02d4d77fdfb93 Faulting
application path: d:\windows\system32\inetsrv\w3wp.exe Faulting module
path: D:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Report
Id: f5e4d6dc-9940-11e4-80bd-0003ff508f5b Faulting package full name:
Faulting package-relative application ID: Event Xml:
1000
2
100
0x80000000000000
467
Application
RD0003FF508F5B
w3wp.exe
8.5.9600.16384
5215df96
clr.dll
4.0.30319.34014
52e0b86c
c00000fd
0000000000195499
c60
01d02d4d77fdfb93
d:\windows\system32\inetsrv\w3wp.exe
D:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
f5e4d6dc-9940-11e4-80bd-0003ff508f5b
Easy steps to reproduce this error:
Download this sample project: http://www.asp.net/mvc/overview/getting-started/getting-started-with-ef-using-mvc/migrations-and-deployment-with-the-entity-framework-in-an-asp-net-mvc-application
Add a CloudServiceProject to the solution and add the web project as a web role
Publish it to azure
Browse the web site and go to students tab (this hits the database and the initializer tries to execute the migrations).
Happens the same.
This is identified as a bug in Microsoft Visual Studio 2013 Update 4. As a temporary work around disable "Lazy Initialization" under IntelliTrace Settings -> IntelliTrace Events. We are investigating fixing this bug in a future update for Visual Studio 2013.
The responsible for the StackOverflowException turned out to be IntelliTrace. I don't know the root cause of the issue between IntelliTrace and Code First Migrations that cause this behavior but when I downloaded the IntelliTrace logs I could see a lot of Sql exceptions:
I have already lost a lot of time with this so I won't investigate it further. The easiest workaround for me now is just disable IntelliTrace before deploying my Web Roles:
If somebody is curious, here are the IntelliTrace logs.
I hope this helps someone else solve such a problem without losing as much time as I did.

NServiceBus MSMQ Send question

I have trouble sending a message via NServiceBus. I have an ASP.Net MVC web app, developing on Win7 x64, I have configured my web.config as
<MsmqTransportConfig InputQueue="worker" ErrorQueue="error" NumberOfWorkerThreads="1" MaxRetries="5" />
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="PricingInformation.Messages" Endpoint="worker2" />
</MessageEndpointMappings> </UnicastBusConfig>
In application_start I wire up the following:
var bus = NServiceBus.Configure.WithWeb()
.StructureMapBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(false)
.PurgeOnStartup(false)
.UnicastBus()
.ImpersonateSender(false)
.CreateBus()
.Start();
When the action I'm interested happens in the app I fire
public override void HandleEvent(SupplierPricingUpdatedEvent updatedEvent)
{
bus.Send(new ModelSupplierDetailsUpdatedMessage() {Id = updatedEvent.Id})
return;
}
ModelSupplierDetailsUpdatedMessage is simple class in PricingInformation.Messages using interface marker IMessage and decorated with Serializable attribute.
The MSMQ queues are setup, not transactional, and everyone including NETWORK SERVICE and IIS_IUSRS have full control (in deseperate troubleshooting measures)
log4net shows the following:
DEBUG NServiceBus.Utils.MsmqUtilities 14 - Checking if queue exists: worker.
DEBUG NServiceBus.Utils.MsmqUtilities 14 - Checking if queue exists: error.
DEBUG NServiceBus.Unicast.UnicastBus Worker.15 - Calling 'HandleBeginMessage' on NServiceBus.SagaPersisters.NHibernate.NHibernateMessageModule
INFO NServiceBus.Unicast.UnicastBus Worker.15 - worker initialized.
DEBUG NServiceBus.Unicast.UnicastBus Worker.15 - Calling 'HandleEndMessage' on NServiceBus.SagaPersisters.NHibernate.NHibernateMessageModule
DEBUG NServiceBus.Unicast.UnicastBus 9 - Sending message PricingInformation.Messages.ModelSupplierDetailsUpdatedMessage, PricingInformation.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null with ID 2c642672-1bf1-48d4-a90f-734e2fdd726d\8267 to destination worker2.
Yet no matter what I try, and what I tweak (been three hours at it already) the message never appears in the queue. I cant find an exception and i have debug level logging on everything.. Its probably something simple help
The problem is that if you manually set up your queues as non-transactional, then it won't work. Setting the NServiceBus "IsTransactional" property to false doesn't mean you can work with non-transactional queues.
Please try deleting the queues and recreating them as transactional or, if you're using the beta of v2.0, just letting NServiceBus create the queues for you.