Suddenly getting ##[error]System.ArgumentNullException: Value cannot be null. (Parameter 'input') - azure-devops

Problem appeared just today - previously it was working fine. Suddenly getting the error:
##[error]System.ArgumentNullException: Value cannot be null. (Parameter ‘input’)
at System.Text.RegularExpressions.Regex.Replace(String input, String replacement)
at Microsoft.VisualStudio.Services.Agent.Util.StringUtil.DeactivateVsoCommands(String input)
at Microsoft.VisualStudio.Services.Agent.Worker.WorkerUtilities.DeactivateVsoCommandsFromJobMessageVariables(AgentJobRequestMessage message)
at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args)
Error reported in diagnostic logs. Please examine the log for more details.
- /home/vsts/agents/2.213.1/_diag/Worker_20221110-110312-utc.log
Pool: Azure Pipelines
Image: ubuntu-latest
Queued: Today at 13:00 [manage parallel jobs]
The agent request is already running or has already completed.
The stage seems waiting for the agent and after that fails with above message.
After cancelling the pipeline and restarting the stage - the same error. What is the problem? What would be temporary workaround?
Our pipelines are defined with yaml.
I have no access to the log indicated above so can not provide more details from there.
Regards,
Roman.
UPDATE 14/11/2022:
After applying patch 2.213.2 to MS hosted agents Microsoft has resolved the issue

===============================================
Update 11/14
According to this latest release for the agent, the error has been fixed in the version of 2.213.2.
You could try to update again and test to see if the issue persists.
If the issue is still observed, you could share the latest status with us.

This issue started yesterday also for us. It's indeed related to the fact the pipeline is using Subversion source. We mitigated the issue by switching to our own private/custom agent pool. Moved away from the public azure devops agents.
Update 1:
The mitigation worked for a few hours but all pipelines are now failing with:
##[error]System.ArgumentOutOfRangeException: The startIndex argument must be greater than or equal to zero. (Parameter 'startIndex')
at System.Collections.Concurrent.ConcurrentStack`1.ValidatePushPopRangeInput(T[] items, Int32 startIndex, Int32 count)
at System.Collections.Concurrent.ConcurrentStack`1.PushRange(T[] items, Int32 startIndex, Int32 count)
at System.Collections.Concurrent.ConcurrentStack`1.PushRange(T[] items)
at Agent.Plugins.Repository.SvnCliManager.<>c__DisplayClass3_0.<GetSvnWorkingCopyPaths>b__0(String fld)
at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey)
at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork()
at System.Linq.Parallel.SpoolingTaskBase.Work()
at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
at System.Linq.Parallel.QueryTask.<>c.<.cctor>b__10_0(Object o)
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

Had the same issue that started yesterday 11/10/2022 with subversion and Azure Hosted Build Agents. Had to build an on premise hosted agent and point build pipeline to use that instead of the azure hosted one.

Related

Visual Studio Test task fails randomly without error message, Exit code 1 returned from process: node.exe

We have this issue that has been plaguing our engineering group for over a year and we cannot make any progress on the resolution. This question on Stack Overflow is eerily similar to what we experience.
We have several pipelines that use the "Visual Studio Test" task to run a set of NUnit tests from a single assembly. There are 1400-1500 unit tests in the project and it takes The NUnit Test Adapter is deployed via the NuGet task (install NUnit3TestAdapter). This test assembly is built with the following settings:
MSBuild Architecture=x86
Platform=AnyCPU
Configuration=Release
Strong Name Signed
Across multiple pipelines, this step randomly yet consistently ends early with zero error message or trace of why the process ended. Running a subsequent build of the pull request will often result in a success but occasionally another failure. In this case we have coined this a "flaky build" where given the same exact code inputs, the build can fail and then subsequently pass.
This is the only error message we received from the pipeline:
##[error]Exit code 1 returned from process: file name 'REDACTED\externals\node\bin\node.exe', arguments '"REDACTED_work_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.170.1\runvstest.js"'.
Finishing: Integration Tests
These tests are our "integration tests" that perform many setup steps to create a functional version of our system including creating a web server via Kestrel to host our API, connect to (localdb) running on the build server to restore a database .bak file and apply a database project (dacpac) as well as connect to Redis (also installed locally on the build server).
We are struggling to determine next steps so I am reaching out to the community to see if anyone else has suffered through this issue, has ideas for how to debug the issue or perhaps even a solution!
edit on 2020-11-10
Yesterday I set the pipeline variable system.debug to true and this is the output it captured during one of our flaky builds.
##[error\]Exit code 1 returned from process: file name 'E:\\servername\\externals\\node\\bin\\node.exe', arguments '"E:\\servername\\\_work\\\_tasks\\VSTest\_ef087383-ee5e-42c7-9a53-ab56c98420f9\\2.170.1\\runvstest.js"'.
##[debug\]Microsoft.VisualStudio.Services.Agent.Util.ProcessExitCodeException: Exit code 1 returned from process: file name 'E:\\servername\\externals\\node\\bin\\node.exe', arguments '"E:\\servername\\\_work\\\_tasks\\VSTest\_ef087383-ee5e-42c7-9a53-ab56c98420f9\\2.170.1\\runvstest.js"'.
at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary\`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue\`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary\`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue\`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary\`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.NodeHandler.RunAsync()
at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync()
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)

Turning on email notifications breaks MS Release Management

I am running TFS 2013 Update 4, Release Management Client Update 4, Release Management Server Update 4, and Update 4 Deployment Agents. I am using ReleaseTfvcTemplate.12.xml.
When a developer checks in code, TFS Build compiles the code, and if it completes then it is released to the DEV stage. This works fine.
However, turning on emails creates a problem.
Let's say I need to notify 10 people of a deployment and then send those same 10 people "approval" emails after the deployment is accepted, which it automatically is. That's 20 emails.
I turned on verbose logging on the RM server and I see that each email takes 30 seconds to send. They send one at a time, one after the other. So it takes ten minutes to send twenty emails.
The emails start sending as soon as the deployment starts. The actual deployment usually takes around 1 minute. Release Management marks the build as deployed and keeps sending the "deploying" and "approval" emails. Meanwhile the TFS Build Configuration log is stuck waiting at:
Process each ConfigurationsToRelease
Release the build
Run the Release Management build process for the current configruation
If a deployment finishes its' emails because they are turned off or there are only 3-4 to send, then the TFS Build Configuration log completes the release and the build is marked successful. However, TFSBuild will only wait 5 minutes at the "Release the build" part of the ReleaseTfvcTemplate workflow. If it takes longer than 5 minutes to send 20 emails, which it does, the build fails. How do I increase this timeout? I have upped the timeout on every component/tool I could find in Release Management. I even changed some web.config timeout settings.
The end result is I end up with deployed code, Release Management thinks everything went fine, and TFS Build thinks the build failed.
Edit:
Here are some lines I pulled from the verbose RM server logs. Notice the timestamps. (I cut some lines out)
7/28/2015 3:49:48 PM - Verbose - (13008, 12024) - A workflow execution
is completed. 7/28/2015 3:49:48 PM - Information - (13008, 12024) -
DeploymentControllerServiceProcessor.OnActivityComplete: Workflow
completed successfully, accept the deployment step. LocalReleaseId:
596, LocalReleaseStepId: 2158 7/28/2015 3:54:47 PM - Information -
(13008, 6952) -
DeploymentControllerServiceProcessor.PrepareNotificationForDeployerImplementation:
NextActivityReadyForDeployment: 7/28/2015 3:54:47 PM - Information -
(13008, 6952) -
DeploymentControllerServiceProcessor.GetNextComponentReadyForDeployment:
DeploymentEvent: 7/28/2015 3:54:49 PM - Information - (13008, 12024)
- Exception in DeploymentControllerServiceProcessor.OnActivityComplete, app.Completed
7/28/2015 3:54:49 PM - Verbose - (13008, 12024) - The request was
aborted: The request was canceled.: \r\n\r\n at
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at
Microsoft.TeamFoundation.Release.Data.WebRequest.PlatformHttpClient.EndGetResponse(IAsyncResult
asyncResult) at
Microsoft.TeamFoundation.Release.Data.WebRequest.RestClientResponseRetriever.EndGetAsyncMemoryStreamFromResponse(IAsyncResult
asyncResult, IPlatformHttpClient platformHttpClient) at
Microsoft.TeamFoundation.Release.Data.WebRequest.RestClientResponseRetriever.EndDownloadString(IAsyncResult
asyncResult, IPlatformHttpClient platformHttpClient) at
Microsoft.TeamFoundation.Release.Data.WebRequest.RestClient.EndPost(IAsyncResult
asyncResult) at
Microsoft.TeamFoundation.Release.Data.Proxy.RestProxy.HttpRequestor.<>c__DisplayClass1.b__0(String
url, String body) at
Microsoft.TeamFoundation.Release.Data.Proxy.RestProxy.BaseNotificationServiceProxy.SendNotification(Int32
releaseId, String releaseName, String applicationVersionName, String
stageTypeName, String environmentName, Int32 releaseStepId, Int32
releaseStepTypeId, Boolean releaseStepIsAutomated) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.CreateNextReleaseStep(Release
release, Stage stage, StageStep stageStep, Int32 releaseStageRank,
Int32 trialNumber) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.MoveToNextReleaseStep(Release
release, Stage currentStage, ReleaseStep currentReleaseStep) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.MoveWorkflowForward(Release
release, ReleasePath releasePath, Stage currentStage, ReleaseStep
currentReleaseStep, Int32 lastStepRankOfCurrentStage) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.AcceptStep(Release
release, Int32 releaseStepId, Int32 actualApproverId, String
approverComment, Nullable1 deferredDateTime) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.CreateNextReleaseStep(Release
release, Stage stage, StageStep stageStep, Int32 releaseStageRank,
Int32 trialNumber) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.MoveToNextReleaseStep(Release
release, Stage currentStage, ReleaseStep currentReleaseStep) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.MoveWorkflowForward(Release
release, ReleasePath releasePath, Stage currentStage, ReleaseStep
currentReleaseStep, Int32 lastStepRankOfCurrentStage) at
Microsoft.TeamFoundation.Release.Workflow.Services.ReleaseWorkflowService.AcceptStep(Release
release, Int32 releaseStepId, Int32 actualApproverId, String
approverComment, Nullable1 deferredDateTime) at
Microsoft.TeamFoundation.Release.ServiceProcessor.Processor.DeploymentControllerServiceProcessor.OnActivityComplete(String
workflow, WorkflowApplicationCompletedEventArgs e)
There is a setting on the "Administration" tab under "Settings" for "TFS Trigger Deployment Timeout". If you increase that, the build won't fail after 5 minutes.
I'd invest some time looking at why it takes 30 seconds to send each email, though. I've never seen that particular problem... it could be a network issue, or an issue with your mail server.

Mongodb worker role throws System.IO.FileNotFoundException

I have an Azure cloud project which makes use of mongodb worker roles. When running in the local emulator everything works fine, however when I deploy to an Azure staging area, the mongodb worker roles are stuck cycling and show a System.IO.FileNotFoundException. I connected to one of the virtual machines remotely and looked at the event viewer and found the following error:
An unhandled exception occurred. Type: System.IO.FileNotFoundException
Process ID: 1936 Process Name: WaWorkerHost Thread ID: 4 AppDomain
Unhandled Exception for role MongoDB.WindowsAzure.MongoDBRole_IN_0
Exception: Could not load file or assembly 'mswacdmi, Version=1.7.0.0,
Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its
dependencies. The system cannot find the file specified. at
Microsoft.WindowsAzure.StorageClient.CloudDrive..ctor(Uri uri,
StorageCredentials credentials) at
Microsoft.WindowsAzure.StorageClient.CloudStorageAccountCloudDriveExtensions.CreateCloudDrive(CloudStorageAccount
storageAccount, String pageBlobUri) at
MongoDB.WindowsAzure.MongoDBRole.Utilities.GetMountedPathFromBlob(String
localCachePath, String cloudDir, String containerName, String
blobName, Int32 driveSize, CloudDrive& mongoDrive) in
c:\Users\Matt\Software\AzurePrototype1\mongo-azure-master\src\MongoDB.WindowsAzure.MongoDBRole\Utilities.cs:line
78 at
MongoDB.WindowsAzure.MongoDBRole.MongoDBRole.GetMongoDataDirectory()
in c:\Users\Matt\Software\AzurePrototype1\mongo-azure-master\src\MongoDB.WindowsAzure.MongoDBRole\MongoDBRole.cs:line 255 at MongoDB.WindowsAzure.MongoDBRole.MongoDBRole.StartMongoD()
in c:\Users\Matt\Software\AzurePrototype1\mongo-azure-master\src\MongoDB.WindowsAzure.MongoDBRole\MongoDBRole.cs:line 201 at MongoDB.WindowsAzure.MongoDBRole.MongoDBRole.OnStart() in
c:\Users\Matt\Software\AzurePrototype1\mongo-azure-master\src\MongoDB.WindowsAzure.MongoDBRole\MongoDBRole.cs:line 91 at
Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType
roleTypeEnum) at
Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.b__0()
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
ignoreSyncCtx) at
System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state) at
System.Threading.ThreadHelper.ThreadStart()
Why does this work locally and not on Azure? I don't understand much about assemblies and such, but I am learning as I go along. One of the things suggested online is to check the properties of references assemblies and set 'Copy Local' to true, I have tried this but still the same error.
In your worker role project, Make sure you have a .NET reference to mswacdmi.dll with the Properties set Copy Local=true. When you publish - RDP to your Azure role instance to verify that your deployed role has the assembly in the /bin directory.
Also make sure you are targeting x64 - I found mswacdmi.dll at this path (Azure SDK 1.8):
C:\Program Files\Microsoft SDKs\Windows Azure.NET SDK\2012-10\bin\runtimes\storage\cloud\x64

Error publishing an asp 3.5 app

Well, I'm here to ask about some problem I have, the detail is that when I publish the application from my computer to the server folder and after make the changes in IIS when I try to browse the web app I get the next error:
Index and length must refer to a location within the string.
Parameter name: length
In my local computer, I installed Visual Studio 2008 with SP1 and in the server is not installed only Framework 3.5.
Is the error related with Entity Framework or it's another problem?
Well, I hope you could help me... regards !
Stack Trace:
[ArgumentOutOfRangeException: Index and length must refer to a location within the string.
Parameter name: length]
System.String.InternalSubStringWithChecks(Int32 startIndex, Int32 length, Boolean fAlwaysCopy) +7495179
System.String.Substring(Int32 startIndex, Int32 length) +11
InnovaWebSite.Global.Session_Start(Object sender, EventArgs e) in C:\Users\cvelazquezt\Desktop\InnovaWeb\InnovaWebSite\Global.asax.cs:117
System.Web.SessionState.SessionStateModule.RaiseOnStart(EventArgs e) +8783312
System.Web.SessionState.SessionStateModule.CompleteAcquireState() +237
System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +504
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +66
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155
The error thrown is from a substring operation. Where it is actually being thrown, I have no clue, as you have not actually posted the stack trace on the error. This could be because you don't know how to do this?
If you can find the stack trace, or even detail your code that gets pieces of a string, you will have a better context of the exception and be able to start doing something to cure it.
As for root cause, the fact it is only happening after publish indicates a) something is missing on the server or b) the server is configured incorrectly (wrong version of .NET for the app pool for example) or c) you have not correctly published the site (maybe copied from your compiled application rather than published and moved the published version over, etc.???)

Re-hydrated workflow throws Profile cannot be null exception

We have a workflow as webservice application. We use custom tracking profile for each workflow type.
The worklfow assembly version got changed, so I changed the profile version too. So as the older workflows should work fine.
Now I started the workflow runtime , a workflow got triggered and persisted to DB
After some time , i re-start the runtime, the workflow is re-hydrated, but try to invoke an event on workflow, it shows exception.
If The workflow engine is not re-started , everything works fine, no errors and workflow is completed.
System.Workflow.Activities.EventDeliveryFailedException: Event "Event_111" on interface type "Service.IService" for instance id "4d0b7397-4ce1-49c7-92c6-92405caa8fe4" cannot be delivered. ---> System.ArgumentNullException: Value cannot be null.
Parameter name: profile
at System.Workflow.Runtime.RTTrackingProfile..ctor(TrackingProfile profile, Activity root, Type serviceType)
at System.Workflow.Runtime.TrackingProfileManager.CreateProfile(TrackingProfile profile, Type workflowType, Type serviceType)
at System.Workflow.Runtime.TrackingProfileManager.GetProfile(TrackingService service, Activity workflow, Version versionId)
at System.Workflow.Runtime.TrackingListenerFactory.GetChannels(Activity schedule, WorkflowExecutor exec, Guid instanceID, Type workflowType, TrackingListenerBroker& broker)
at System.Workflow.Runtime.TrackingListenerFactory.GetListener(Activity sked, WorkflowExecutor skedExec, TrackingListenerBroker broker)
at System.Workflow.Runtime.TrackingListenerFactory.GetTrackingListener(Activity sked, WorkflowExecutor skedExec, TrackingListenerBroker broker)
at System.Workflow.Runtime.TrackingListenerFactory.WorkflowExecutorInitializing(Object sender, WorkflowExecutorInitializingEventArgs e)
at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
at System.Workflow.Runtime.WorkflowRuntime.WorkflowExecutorCreated(WorkflowExecutor workflowExecutor, Boolean loaded)
at System.Workflow.Runtime.WorkflowExecutor.ReRegisterWithRuntime(WorkflowRuntime workflowRuntime)
at System.Workflow.Runtime.WorkflowRuntime.RegisterExecutor(Boolean isActivation, WorkflowExecutor executor)
at System.Workflow.Runtime.WorkflowRuntime.Load(Guid key, CreationContext context, WorkflowInstance workflowInstance)
at System.Workflow.Runtime.WorkflowRuntime.GetWorkflow(Guid instanceId)
at System.Workflow.Activities.WorkflowMessageEventHandler.EventHandler(Object sender, ExternalDataEventArgs eventArgs)
--- End of inner exception stack trace ---
at System.Workflow.Activities.WorkflowMessageEventHandler.EventHandler(Object sender, ExternalDataEventArgs eventArgs)
at RecipeChangeService.RecipeChangeService.MfgEngOrTLApproved(Guid instanceId, ResponseDataObject rdo) in E:\MES\trunk\DotNet\WorkflowDesignProject\WorkflowDesignProject\Workflow Types\RecipeChangeWF\RecipeChangeService.cs:line 64
Any ideas, what is getting wrong? There are two workflows of different versions, but attached to same profile version, can this be issue?
I found the solution for it. The version of tracking profile should match the version while creating tracking profile xml.
TrackingProfile myProfile = new TrackingProfile();
myProfile.Version = new Version("4.0.0.0");
The verison should be same as the version used in stored procedure "UpdateTrackingProfile"
This solved my problem.