I have two main symptoms of a larger problem.
If I'm debugging an application in visual studio every once in a while I'll have it on a breakpoint and the service will just close. Here is the log when it "crashes"
WARNING: Failed to contact Naming Service. Attempting to contact Failover Manager Service...
WARNING: Failed to contact Failover Manager Service, Attempting to contact FMM...
WARNING: Failed to contact Failover Manager Service, Attempting to contact FMM...
Connect-ServiceFabricCluster : Could not ping any of the provided Service Fabric gateway endpoints.
At line:1 char:8
+ [void](Connect-ServiceFabricCluster); Import-Module 'C:\Program Files ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Connect-ServiceFabricCluster], FabricTransientException
+ FullyQualifiedErrorId : TestClusterConnectionErrorId,Microsoft.ServiceFabric.Powershell.ConnectCluster
WARNING: Unable to Verify connection to Service Fabric cluster.
Test-ServiceFabricClusterConnection : Cluster connection instance is null
At C:\Program Files\Microsoft SDKs\Service
Fabric\Tools\PSModule\ServiceFabricSDK\Get-ServiceFabricApplicationStatus.ps1:27 char:16
+ [void](Test-ServiceFabricClusterConnection)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (:) [Test-ServiceFabricClusterConnection], NullReferenceException
+ FullyQualifiedErrorId : GetClusterConnectionErrorId,Microsoft.ServiceFabric.Powershell.TestClusterConnection
Finished executing script 'Get-ServiceFabricApplicationStatus'.
Time elapsed: 00:02:04.7891918
The program '[27216] program.exe' has exited with code 0 (0x0).
Connect-ServiceFabricCluster : Could not ping any of the provided Service Fabric gateway endpoints.
At line:1 char:8
+ [void](Connect-ServiceFabricCluster); Import-Module 'C:\Program Files ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Connect-ServiceFabricCluster], FabricTransientException
+ FullyQualifiedErrorId : TestClusterConnectionErrorId,Microsoft.ServiceFabric.Powershell.ConnectCluster
WARNING: Unable to Verify connection to Service Fabric cluster.
Test-ServiceFabricClusterConnection : Cluster connection instance is null
At C:\Program Files\Microsoft SDKs\Service
Fabric\Tools\PSModule\ServiceFabricSDK\Get-ServiceFabricApplicationStatus.ps1:27 char:16
+ [void](Test-ServiceFabricClusterConnection)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (:) [Test-ServiceFabricClusterConnection], NullReferenceException
+ FullyQualifiedErrorId : GetClusterConnectionErrorId,Microsoft.ServiceFabric.Powershell.TestClusterConnection
Finished executing script 'Get-ServiceFabricApplicationStatus'.
Time elapsed: 00:02:04.7166263
After maybe an hour or two of having service fabric open the cluster becomes unresponsive and I'm unable to deploy any application until I reset the cluster. The logs just constantly say:
Started executing script 'Get-ServiceFabricApplicationStatus'.
Finished executing script 'Get-ServiceFabricApplicationStatus'.
I am also unable to open the service fabric manager via the icon in the task bar but the url http://localhost:19080/Explorer still works.
Running: C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\CleanCluster.ps1 seems to reset it just fine and it works again for a couple of hours. Until (presumably) it gets to a point below where it runs out of connections and everything goes wrong.
The only logs I can seem to find on this are:
thousands of lines of this:
localhost:19000-127.0.0.1:59598: failed to locate connection 1eedf052ba0 to update instance
exceeded client connection limit: target=127.0.0.1:59598 max=1000
localhost:19000-127.0.0.1:59597: failed to locate connection 1eedf055db0 to update instance
Warning 2/26/2018 12:27:11 PM Microsoft-Service Fabric 57089 EntreeServiceProxy
Warning 2/26/2018 12:27:11 PM Microsoft-Service Fabric 4097 Transport
exceeded client connection limit: target=127.0.0.1:59596 max=1000
localhost:19000-127.0.0.1:59595: failed to locate connection 1eedf0536c0 to update instance
exceeded client connection limit: target=127.0.0.1:59595 max=1000
exceeded client connection limit: target=127.0.0.1:59595 max=1000
exceeded client connection limit: target=127.0.0.1:59594 max=1000
localhost:19000-127.0.0.1:59593: failed to locate connection 1eedf052ba0 to update instance
exceeded client connection limit: target=127.0.0.1:59593 max=1000
And my dns service seems to be in an unhealthy state:
Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus',
HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures during open on _Node_0. -2147467259
For more information see: http://aka.ms/sfhealth
Any advice on how to troubleshoot this service fabric issue?
Related
I have created a Service Fabric cluster in Azure portal by following below url.
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-creation-via-portal
After creating the Service fabric cluster I am not able to browse the "Client connection endpoint" or "Service Fabric Explorer". I am getting Access denioed - HTTP ERROR 403.
I also installed the default certificate on my machine which is creating at the time of Service fabric cluster creation.
When I tried to publish from visual studio I got below errors
Connect-ServiceFabricCluster : Cannot bind parameter 'StoreLocation'. Cannot convert value "" to type
"System.Security.Cryptography.X509Certificates.StoreLocation". Error: "The identifier name cannot be processed
because it is either too similar or identical to the following enumerator names: CurrentUser, LocalMachine. Use a more
specific identifier name."
At C:\CODE\DotNetCode\SFSample\SFSample\Scripts\Deploy-FabricApplication.ps1:185 char:45
... [void](Connect-ServiceFabricCluster #ClusterConnectionParameters)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Connect-ServiceFabricCluster], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.ServiceFabric.Powershell.ConnectCluster
WARNING: Unable to verify connection to Service Fabric cluster.
Test-ServiceFabricClusterConnection : Cluster connection instance is null
On you SF project, you have a Publish profile called Cloud.xml
You should set the configuration of your cluster in there to tell the thumbprint of the certificate and where you stored it.
Something like:
<?xml version="1.0" encoding="utf-8"?>
<PublishProfile xmlns="http://schemas.microsoft.com/2015/05/fabrictools">
<ClusterConnectionParameters ConnectionEndpoint="mycluster.westus.cloudapp.azure.com:19000"
X509Credential="true"
ServerCertThumbprint="0123456789012345678901234567890123456789"
FindType="FindByThumbprint"
FindValue="9876543210987654321098765432109876543210"
StoreLocation="CurrentUser"
StoreName="My" />
<ApplicationParameterFile Path="..\ApplicationParameters\Cloud.xml" />
<CopyPackageParameters CompressPackage="true" />
</PublishProfile>
You probably didn't configure it or did in incorrect.
We got a Service Fabric cluster in our development environment using 2 VMs. I was trying to upgrade the Application deployed in SF using the following command:
Start-ServiceFabricApplicationUpgrade -ApplicationName "fabric:/ApplicationName" -ApplicationTypeVersion "3.7.2625.0" -UnMonitoredAuto
I get the following error as a result:
Start-ServiceFabricApplicationUpgrade : Application type and version not found
At line:1 char:1
+ Start-ServiceFabricApplicationUpgrade -ApplicationName "fabric:/ApplicationName" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (Microsoft.Servi...usterConnection:ClusterConnection) [Start-ServiceFabricApplicationUpgrade], FabricElementNotFoundException
+ FullyQualifiedErrorId : UpgradeApplicationErrorId,Microsoft.ServiceFabric.Powershell.StartApplicationUpgrade
I would like to know whether there are any configurations I need to change at the ClusterConfiguration level. Any help would be appreciated.
Thanks.
There are 3 simple steps to upgrade an application:
Copy-ServiceFabricApplicationPackage
Register-ServiceFabricApplicationType
Start-ServiceFabricApplicationUpgrade
By the message you posted, the error is likely because you missed the step 2.
If you have executed the step 1, 2 and 3, certify that:
The application package has been registered correctly
The application version you registered is correct, both match, the package and the Upgrade command
The existing application and the registered one are from same type
Check this doc for more info: Service Fabric application upgrade using PowerShell
I am trying to use the SetPassword method on a DirectoryEntry.
$Root = New-Object System.DirectoryServices.DirectoryEntry("LDAP://dc.ourdomain.com/OU=Users,DC=dc,DC=ourdomain,DC=com", $ldapusername, $ldappassword, "Secure")
$objUser = $Root.Create("user","CN=$DisplayName")
$objUser.Put("sAMAccountName", "$username")
$objUSer.Put("userPrincipalName","$userPrincipalName")
$objUser.Put("givenName", "$firstname")
$objUser.Put("sn", "$lastname")
$objUser.Put("description", "$description")
$objUser.Put("displayName", "$DisplayName")
$objUser.Put("mail", "$email")
$objUser.Put("company", "$Company")
$objUser.SetInfo()
$ObjUser.psbase.Invoke("SetPassword","qW12eR34")
$objUser.psbase.InvokeSet(“AccountDisabled”,$false)
$objUser.SetInfo()
Whenever I run this, I keep getting the following error:
Exception calling "Invoke" with "2" argument(s): "The RPC server is unavailable. (
Exception from HRESULT: 0x800706BA)"
At E:\Untitled1.ps1:27 char:23 + $ObjUser.psbase.Invoke <<<< ("SetPassword","qW12eR34")
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodTargetInvocation
I am able to reset the password through Set-ADAccountPassword just fine, but I need to be able to do it through this way.. This function creates the user just fine, and it also sets the AccountDisable to false as it should do.
But it keeps failing on SetPassword..
I read around here that some people mentioned it could be due to blocked ports etc (Everything is open between the two servers) and some mentioned the AuthenticationType has to be set to Secure.. But I am setting it to Secure, and I am still having the same problem.
Thanks in advance..
Ok so i am pretty sure i found out why this is, i was trying to do something the same, and confirmed all ports (this works on port 135), are open, but port 135 never was able to recieve traffic from the world, called my ISP and yes, most ISP's these days, and with more to follow WILL block all traffic incoming on this port, so that's what your error is telling you.
It is trying to perform the actions but the requests are never reaching the server.
Call your provider and confirm, else check port 135 in your firewall rules on your incoming server (temporarily disabling firewall for testing will tell you)
Good morning,
I am using PowerShell to interact with the VMWare vCloud API and am having problems after disconnecting from vCloud.
The process for connecting and using the API is as follows:
Connect to vCloud using a POST
Perform necessary operations
Disconnect using DELETE (This clears down the session at the remote endpoint)
All communication is over HTTPS.
The problem is that when this process run for the first in a new PowerShell console everything is OK. However if I want to repeat this process again I invariably get the following message:
Invoke-WebRequest : The underlying connection was closed: An unexpected error occurred on a send.
At C:\tfs\poshvcloud\solutions\poshvcloud\functions\helpers\_Invoke- vCloudRequest.ps1:95 char:17
+ $response = Invoke-WebRequest #splat
+ ~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
I have used Fiddler in an attempt to find out what is going on, and interestingly I only ever see 'CONNECT' operations and no attempt to even communicate with vCloud when the above error is displayed.
Indeed I have checked the ServiceEndPoint using [System.Net.ServicePointManager]::FindServicePoint("https://vcloud.example.com/") and I get an object back and it states I do not have any connections, but for some reason the connection is not re-established when I try to connect again. It is as if PowerShell does not bother with SSL handshake again.
BindIPEndPointDelegate :
ConnectionLeaseTimeout : -1
Address : https://vcloud.example.com/
MaxIdleTime : 100000
UseNagleAlgorithm : True
ReceiveBufferSize : -1
Expect100Continue : False
IdleSince : 18/07/2014 09:11:34
ProtocolVersion : 1.1
ConnectionName : https
ConnectionLimit : 2
CurrentConnections : 0
Certificate : System.Security.Cryptography.X509Certificates.X509Certificate
ClientCertificate :
SupportsPipelining : True
One weird thing is that if I attempt to do another operation quickly enough it does go through, but then eventually times out and I get the error.
The only way I can this to work again is to close down my current PowerShell console and start a new one and then load all the necessary modules.
I have tried to 'DisableKeepAlive' on the Inovke-WebRequest but this causes the API calls to fail completely. I then tried to just do it on the disconnect but this did not work either.
I am thinking that I might need to write my own version Invoke-WebRequest, but I still think I am going to have to clear down things and I not sure how. I would rather stick with Invoke-WebRequest if I can.
I know that there is PowerCLI from VMWare that takes care of all of this, however that is a huge module and only a fraction of it is applicable to vCloud. Indeed I started off using it, but due to some inconsistencies and a large dependency I have created my own vCloud module talking to the vCloud REST API directly.
I hope this is enough information, but please let me know if more is required. Any help is greatly appreciated.
Kind regards, Russell
$result = Invoke-WebRequest $adress -Method $Method
$result.BaseResponse.Close()
I'm trying to use the Use-AzureHDInsightCluster powershell cmdlet
Use-AzureHDInsightCluster "mycluster" -Subscription (Get-AzureSubscription -Current).SubscriptionId
but I get the following error. Not sure where to start as I'm new to this.
Use-AzureHDInsightCluster : An error occurred while sending the request.
At line:1 char:1
+ Use-AzureHDInsightCluster "mycluster" -Subscription (Get-AzureSubscription -Cu ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Use-AzureHDInsightCluster], HttpRequestException
+ FullyQualifiedErrorId : System.Net.Http.HttpRequestException,Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.
PSCmdlets.UseAzureHDInsightClusterCmdlet
EDIT I've added the debug flag and I get a little more information
DEBUG: Severity: Error
One or more errors occurred.
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.PSCmdlets.UseAzureHDInsightClusterCmdlet.EndProcessing()
An error occurred while sending the request.
at System.Threading.Tasks.TaskAwaiter`1.GetResult()
at
Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.Commands.CommandImplementations.UseAzureHDInsightClusterCommand.<End
Processing>d__0.MoveNext()
Unable to connect to the remote server
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
An attempt was made to access a socket in a way forbidden by its access permissions 70.37.71.239:8443
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket,
IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
Remove WindowsAzureProfile.xml from below folder and try again:
C:\Users\xxxx\AppData\Roaming\Windows Azure Powershell
The error message is not very helpful, isn't it. PowerShell has some common switches. Two of them are helpful for debugging. They are -debug and -verbose. Retry the command with the two switch and get the detailed information.
Is "mycluster" a valid cluster name? Is it a name you used for this post? Can you test the Windows Azure connection using the following commands:
Get-AzureSubscription
Get-AzureHDInsightCluster
If you don't receive your subscriptions nor your cluster list. You might need to fix your connection first. For information on connecting to your Azure account, see http://www.windowsazure.com/en-us/manage/install-and-configure-windows-powershell/.