TFS Build Agent stopped - build-server

I have a problem with a running build machine, where the agent suddenly does not want to start. It's been a part of a remote controller and for trouble shooting this issue, if started a local controller. The symptoms are, at the agent(s) initializes correct (says 'Ready'), but has the stopped icon and in the status area says 'BuildController has not been started in 1 minutes. The AD account running the build service works on another build machine (seperate controller + build agents). I've tried the following
Reinstall the build service
running with machine name, fully qualified domain name and IP address for endpoint address
un- and re-registered build service
rebooted
cleaned up build agent registrations with script
If I change the service account running the build service to my own AD account, it works. However, running under our dedicated build user failes on this particular machine, but not the other. Any suggestions what to do? Here's the error from the event log:
Service 'Default Agent - tfs2010build1' had an exception:
Exception Message: There was no endpoint listening at http://tfs2010build1:9191/Build/v3.0/Services/Controller/31 that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. (type EndpointNotFoundException)
Exception Stack Trace:
Server stack trace:
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.TeamFoundation.Build.Machine.IBuildControllerService.TestConnectionFromController(String agentUri)
at Microsoft.TeamFoundation.Build.Machine.ServiceProxies.ServiceProxy`1.<>c__DisplayClass3.<Do>b__2(T channel)
at Microsoft.TeamFoundation.Build.Machine.ServiceProxies.ServiceProxy`1.Do[TResult](Func`2 action)
at Microsoft.TeamFoundation.Build.Machine.BuildAgentService.<>c__DisplayClass12.<TestConnection>b__11(Object )
Inner Exception Details:
Exception Message: Unable to connect to the remote server (type WebException)
Exception Stack Trace: at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
Inner Exception Details:
Exception Message: No connection could be made because the target machine actively refused it 127.0.0.1:38742 (type SocketException)
Exception Stack Trace: at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

Found the problem. Proxy server was enabled under IE options. Not sure why the build service worked under my AD user account and not the dedicated build user, but it solved the problem.
---->>>>>Update!
So we have 2 machines (B1 & B2), each with 2 agents. B1 had the initial problem and was solved by disabling the proxy settings under IE. Yesterday B2 suddenly started showing the same symptons and error messages on the 2 agents. Proxy setting is NOT enabled. While it did fix B1, it's not the universal solution for this particular problem.
It's hard work keeping these build agents running :( - Miss TeamCity...
---->>>>Update again!
So yesterday when I looked at the proxy configuration, it wasn't set. However this morning the checkbox was checked. Disabled the proxy and the agents went online. Very strange behavior! Wonder if Windows Update changes these settings...

I often get this same issue with the proxy when I am forced to manually stop a build. I have not been able to find any decent resolutions for this.

Related

Azure Devops Pipeline Environment Resource Agent installation issue

I have been trying to add an environment resource to our pipeline in azure devops but every time i install it with the powershell script they provide it asks me for tags. If i don't want to input tags it errors. If i choose to put in tags it errors. Either way it starts over every time and it will never successfully install. Anyone know if this is something on my end or on their end? I looked through the diagnostic file and the exceptions it is logging look like it's something that i can't fix but i have been having issues with this for over a week and thinking it was something that Microsoft would realize and patch. Now i'm wondering if there is something else i can do?
[2020-05-22 13:01:48Z ERR VisualStudioServices] POST request to https://DEVOPS_URL/24cca667-60da-4ba2-a323-4e05c46f3309/_apis/pipelines/environments/3/providers/virtualmachines failed. HTTP Status: InternalServerError, AFD Ref: Ref A: 56C7161B437D41698EBBDE7ACBF4CAA2 Ref B: ATAEDGE0918 Ref C: 2020-05-22T13:01:48Z
[2020-05-22 13:01:48Z INFO CommandSettings] Flag 'unattended': 'False'
[2020-05-22 13:01:48Z ERR Terminal] WRITE ERROR (exception):
[2020-05-22 13:01:48Z ERR Terminal] Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: TF400898: An Internal Error Occurred. Activity Id: acb8a36a-e602-4988-b3f7-8fbeecd729e0.
---> System.NullReferenceException: TF400898: An Internal Error Occurred. Activity Id: acb8a36a-e602-4988-b3f7-8fbeecd729e0.
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable1 queryParameters, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.EnvironmentVMResourceConfigProvider.AddAgentAsync(AgentSettings agentSettings, TaskAgent agent, CommandSettings command)
at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.ConfigureAsync(CommandSettings command)
I was experiencing a similar error "Failed to add virtual machine resource. Linked environment pool is null."
Answer from Kevin Ross here: https://developercommunity.visualstudio.com/t/addition-of-resource-to-environment-fails-for-user/1048111
I resolved my error by following the below steps:
Get the deployment pool ID from the environment URL
Find the deployment pool settings in the organization settings and navigate to the correct pool based on ID from step 1
From the deployment pool settings, go to security and add the required user to administrator role or whatever role is required

Failed to Create Development Service Fabric Cluster on Windows Server 2016 Standard

I am attempting to create a local development (unsecured) Service Fabric Cluster on Windows Server 2016 Standard. I have followed the instructions found in this article. However, I'm getting a rather interesting error and cannot find anything to help me resolve this.
FabricHostSvc was not installed by FabricInstallerSvc on machine
localhost. FabricSetup may have failed. CreateCluster Error:
System.AggregateException: One or more errors occurred. --->
System.Fabric.FabricServiceNotFoundExc eption: FabricHostSvc was not
installed by FabricInstallerSvc on machine localhost. FabricSetup may
have failed. at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(Str
ing machineName, ServiceController installerSvc) at
System.Threading.Tasks.Parallel.<>c__DisplayClass17_01.<ForWorker>b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at
System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object
) --- End of inner exception stack trace --- at
System.Threading.Tasks.Task.ThrowIfExceptional(Boolean
includeTaskCanceledExceptions) at
System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout,
CancellationToken cancellationToken) at
System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive,
Int32 toExclusive, ParallelOptions parallel Options, Action1 body,
Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit,
Action1 localFinally) at
System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1
source, ParallelOptions parallelOption s, Action1 body, Action2
bodyWithState, Action3 bodyWithStateAndIndex, Func4
bodyWithStateAndLocal, Func5 bodyWithE verything, Func1 localInit,
Action1 localFinally) at
System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source,
Action1 body) at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.RunFabricServices(List1
machines, FabricPacka geType fabricPackageType) at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.<CreateClusterAsyncInternal>d__7.MoveNext()
---> (Inner Exception #0) System.Fabric.FabricServiceNotFoundException: FabricHostSvc was not
installed by FabricInstall erSvc on machine localhost. FabricSetup may
have failed. at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.StartAndValidateInstallerServiceCompletion(Str
ing machineName, ServiceController installerSvc) at
System.Threading.Tasks.Parallel.<>c__DisplayClass17_01.b__1()
at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
at
System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object
)<---
Cleaning up faulted installation. FabricRoot not found in registry of
target machine localhost. Create Cluster failed. For more information
please look at traces in FabricLogRoot. Create Cluster failed with
exception: System.AggregateException: One or more errors occurred.
---> System.AggregateExcep tion: One or more errors occurred. at Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManager.d__0.MoveNext()
--- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean
includeTaskCanceledExceptions) at
System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout,
CancellationToken cancellationToken) at
Microsoft.ServiceFabric.Powershell.ClusterCmdletBase.NewCluster(String
clusterConfigurationFilePath, String fabric PackageSourcePath, Boolean
cleanupOnFailure)
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManagerInternal.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Microsoft.ServiceFabric.DeploymentManager.DeploymentManager.d__0.MoveNext()<---
Has anyone encountered this error before and fixed it? How is this error resolved?
Side Note: After receiving this error I ran the CleanFabric PowerShell script and removed all the Service Fabric files from the server and tried running the installation again with the same error message.
In addition, there are no Service Fabric SDKs installed on the machine (the ones you'd use on a local development machine). The reason for this is due to the official prerequisites stated by Microsoft shown below.
Prerequisites for each machine that you want to add to the cluster:
1. A minimum of 16 GB of RAM is recommended.
2. A minimum of 40 of GB available disk space is recommended.
3. A 4 core or greater CPU is recommended.
4. Connectivity to a secure network or networks for all machines.
5. Windows Server 2012 R2 or Windows Server 2012 (you need to
have KB2858668 installed).
6. .NET Framework 4.5.1 or higher, full install.
7. Windows PowerShell 3.0. The RemoteRegistry service should be running on all the machines.
The cluster administrator deploying and configuring the cluster must have administrator privileges on each of the machines. You cannot install Service Fabric on a domain controller.
I cannot help but feel there is something obvious missing but I've followed the docs very closely so this is rather perplexing.
Service Fabric drivers have a signing issue which is preventing them from being installed on Windows Server 2016 and Windows 10 Anniversary edition. Please wait for the next version or try with version 5.2.

System.IO.FileNotFoundException when running TCP Socket client tutorial in Windows Phone 8

I'm trying to run the How to create and use a TCP socket client app for Windows Phone but get the following errors ...
An exception of type 'System.IO.FileNotFoundException' occurred in mscorlib.ni.dll and wasn't handled before a managed/native boundary
An exception of type 'System.Net.Sockets.SocketException' occurred in System.Net.ni.dll and wasn't handled before a managed/native boundary
An exception of type 'System.Net.Sockets.SocketException' occurred in System.Net.ni.dll and wasn't handled before a managed/native boundary
An exception of type 'System.Net.Sockets.SocketException' occurred in System.Net.ni.dll and wasn't handled before a managed/native boundary
An exception of type 'System.Net.Sockets.SocketException' occurred in System.Net.ni.dll and wasn't handled before a managed/native boundary
I have the Simple IP services running on my dev computer and the phone is linked via USB when debugging. It does not suggest in the article what you should use for the IP address to reach the dev computer from the phone - I have tried,
192.168.1.66 // Dev computer address on lan
whitby // Dev computer name
127.0.0.1 // Localhost, refers to dev computer or phone?
localhost // ditto
I'm guessing as well, if the host was wrong I would get a nice error message - instead, nothing (i.e. empty string) is echoed back from e.SocketError.ToString()
I am using Windows Phone 8.
Actually, I've figured it out. I was used to Windows Phone 7 programming and it seems that the System libraries throw exceptions as a matter of course in Windows Phone 8 - the app continues to run.
The problem was that the firewall was blocking the Simple IP services ports. To get the to work you can temporarily disable the Windows Firewall:
Type 'Firewall' at the Start Menu to get the 'Windows Firewall with Advanced Security' client
Click 'Windows Firewall Properties'
Under the 'Private Profile' tab, set the 'Firewall State' to 'Off' temporarily
Then connect using the IP of the dev computer e.g. 192.168.1.66 (not localhost, 127.0.0.1 etc.)

Crystal report 9.2, incorrect log on parameters

Background;
The Web application is developed for .Net framework 4.0. It has crystal report 9.2 integration. Application runs on Integrated Windows Authentication. The crystal reports are working fine when we execute, from solution running under Visual Studio 2010. When the same report are deployed to server (Web Server- OS: Windows Server 2003-SP2-32bit. DB Server- OS: Windows Server 2003-32bit) following error occurs, incorrect log on parameters. Crystal Report Runtime Engine for .net framework 4.0 has been installed in web server. Reports are configured to work on ODBC, System DSN with SQL Server driver. This drivers runs on SQL Server user account which has permission on database. There are no logon parameters passed from application. Just to verify, solution of passing the logon parameters from application is also tried, but the problem is not resolved.
Note: To resolve full access has been provided to IIS_WPG account on folder: C:\Windows\Temp, C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files and web application.
Server Error in '/XXXX' Application.
________________________________________
Error in File E:\WebApps\XXXX\Reports\CompanyStandard.rpt:
Unable to connect: incorrect log on parameters.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: CrystalDecisions.CrystalReports.Engine.LogOnException: Error in File E:\WebApps\XXXX\Reports\CompanyStandard.rpt:
Unable to connect: incorrect log on parameters.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[LogOnException: Error in File E:\WebApps\XXXX\Reports\CompanyStandard.rpt:
Unable to connect: incorrect log on parameters.]
. N(String -, EngineExceptionErrorID 0) +582
. I(Int16 !, Int32 ") +277
CrystalDecisions.CrystalReports.Engine.FormatEngine.GetPage(PageRequestContext reqContext) +429
CrystalDecisions.ReportSource.LocalReportSourceBase.GetPage(PageRequestContext pageReqContext) +172
CrystalDecisions.Web.ReportAgent.|(Boolean Z) +223
CrystalDecisions.Web.CrystalReportViewer.OnPreRender(EventArgs e) +165
System.Web.UI.Control.PreRenderRecursiveInternal() +103
System.Web.UI.Control.PreRenderRecursiveInternal() +175
System.Web.UI.Control.PreRenderRecursiveInternal() +175
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2496
You need to change the profile IIS is running under to a domain account or change your web.config file to impersonate the user. Otherwise you will need to apply logoninfo for the report.

Random COMException on Web App "The handle is invalid"

All of a sudden, I am consistently getting a weird COMException on this machine when I try serve a web app with IIS. I never used to have any problems with this.
Stack Trace:
[COMException (0x80070006): The handle is invalid. (0x80070006 (E_HANDLE))]
[FileLoadException]
System.ServiceModel.Activation.HttpModule.ProcessRequest(Object sender, EventArgs e) +0
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +68
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75
How can I fix this? I need this server to be working later today for a demo.
Things I have tried:
Stop/Start in IIS
aspnet_regiis -i
reboot
It is not actually a COM error, it is a Windows error, wrapped in a COM error code. It is a very low-level error, little can be distilled from the error code or the stack trace. There are plenty of hits when you Google the message, here's one that looks good. Be sure to find more yourself if that's not a good match.