Is it possible to configure startup order when starting up the services.
A Service1 has to be running before Service2 can be started.
Clarification:
I'm didn't mean micro services when I mentioned Service, I meant stateless services like REST API (Service1) and WebSocket (Service2).
So when then solution is deployed the WebSocket service (Service2) must be up and running before the REST API (Service1)?
Of course you can, because you control when services are created. It's not immediately obvious if you've only ever deployed applications through Visual Studio, because Visual Studio sets you up with Default Services. This is what you see in ApplicationManifest.xml when you create an application through Visual Studio:
<DefaultServices>
<Service Name="Stateless1">
<StatelessService ServiceTypeName="Stateless1Type" InstanceCount="[Stateless1_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
<Service Name="Stateful1">
<StatefulService ServiceTypeName="Stateful1Type" TargetReplicaSetSize="[Stateful1_TargetReplicaSetSize]" MinReplicaSetSize="[Stateful1_MinReplicaSetSize]">
<UniformInt64Partition PartitionCount="[Stateful1_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
</StatefulService>
</Service>
</DefaultServices>
This is a nice convenience when you know you always want certain services created a certain way each time you create an application instance. You can define them declaratively here and Service Fabric will create them whenever you create an instance of the application.
But it has some drawbacks. Most notably, in your case, is that you have no control over the order in which the services are created.
It also hides some of the concepts around application and service types and application and service instances, which again can be convenient until you want to do something more advanced, like in your case.
When you "deploy" an application, there are actually several steps:
Create the application package
Copy the package up to the cluster
Register the application type and version
Create an instance of the registered application type and version
Create instances of each registered service type in that application
With Default Services, you skip step 5 because Service Fabric does it for you. Without Default Services though, you get to create your service instances yourself, so you can determine what order to do it in. You can do other things like check if a service is ready before creating the next one. All of these actions are available in Service Fabric's C# SDK and PowerShell cmdlets. Here's a quick PowerShell example:
Copy-ServiceFabricApplicationPackage -ApplicationPackagePath C:\temp\MyApp -ImageStoreConnectionString fabric:ImageStore -ApplicationPackagePathInImageStore MyApp
Register-ServiceFabricApplicationType MyApp
New-ServiceFabricApplication -ApplicationName fabric:/MyAppInstance -ApplicationTypeName MyApp -ApplicationTypeVersion 1.0
New-ServiceFabricService -ApplicationName fabric:/MyAppInstance -InstanceCount 1 -PartitionSchemeSingleton -ServiceName fabric:/MyAppInstance/MyStatelessService -ServiceTypeName MyStatelessService -Stateless
New-ServiceFabricService -ApplicationName fabric:/MyAppInstance -MinReplicaSetSize 2 -PartitionSchemeSingleton -ServiceName fabric:/MyAppInstance/MyStatefulService -ServiceTypeName MyStatefulServiceType -Stateful -TargetReplicaSetSize 3
Of course, this just applies to creating the service instances. When it comes time to upgrading your services, the "upgrade unit" is actually the application, so you can't pick the order in which services within an application get upgraded, at least not in one single upgrade. You can, however, choose which services get upgraded during an application upgrade, so if you have the same ordering dependency, you can accomplish that by doing two separate application upgrades.
So you get some level of control. But, it's really best if your services are resilient to missing dependent services, because there will likely be times when a service is unavailable for one reason or another.
Edit:
I showed you a lot of PowerShell but I should mention the C# APIs are also just as powerful. You have the full set of management tools at your disposal in C#. For example, you can have a service that creates and manages other services. In your case, if Service A depends on Service B, then you can have Service A create an instance of Service B before Service A itself does any work, and throughout Service A's life it can keep an eye on Service B. Here's an example of a service that creates other applications and services: https://github.com/Azure-Samples/service-fabric-dotnet-management-party-cluster/blob/master/src/PartyCluster.ApplicationDeployService/FabricClientApplicationOperator.cs
In service fabric world it's called Service Affinity (https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cluster-resource-manager-advanced-placement-rules-affinity/)
Anyway, try to avoid these situations in microservices world.
Good luck.
Related
Working with a stateful service in ServiceFabric version 5.1.163.9590, I am attempting to deploy a demo application with three WebApi services that manage their own state.
Two of the three services start and create their partitions without errors, but the last spews events a series of warnings and errors, the error detail has this intriguing message:
Microsoft.ServiceFabric.Replicator.LoggingReplicator : GetCopyState The parameter copyContext is null. This might be caused by deployment bug that 'hasPersistedState' attribute is false.
I can't locate any external references to this error message.
Is there a way to correct this from the application and service deployment side, or from the cluster management side?
The error indicates you have a stateful service with persisted state, but didn't tell Service Fabric about that when you deployed the service.
There's a flag that needs to be set to indicate to Service Fabric that a stateful service has persisted state (as opposed to state that is "volatile," meaning in-memory only).
In your ServiceManifest.xml, make sure you have this flag set on the service type:
<ServiceTypes>
<StatefulServiceType ServiceTypeName="MyServiceType" HasPersistedState="true" />
</ServiceTypes>
Then if you're deploying through PowerShell, make sure you set this flag when you create an instance of the service:
PS > New-ServiceFabricService -Stateful -HasPersistedState -ServiceTypeName "MyServiceType" ...
I'm a newbie in Cloud Foundry. In following the reference application provided by Predix (https://www.predix.io/resources/tutorials/tutorial-details.html?tutorial_id=1473&tag=1610&journey=Connect%20devices%20using%20the%20Reference%20App&resources=1592,1473,1600), the application consisted of several modules and each module is implemented as micro service.
My question is, how do these micro services talk to each other? I understand they must be using some sort of REST calls but the problem is:
service registry: Say I have services A, B, C. How do these components 'discover' the REST URLs of other components? As the component URL is only known after the service is pushed to cloud foundry.
How does cloud foundry controls the components dependency during service startup and service shutdown? Say A cannot start until B is started. B needs to be shutdown if A is shutdown.
The ref-app 'application' consists of several 'apps' and Predix 'services'. An app is bound to the service via an entry in the manifest.yml. Thus, it gets the service endpoint and other important configuration information via this binding. When an app is bound to a service, the 'cf env ' command returns the needed info.
There might still be some Service endpoint info in a property file, but that's something that will be refactored out over time.
The individual apps of the ref-app application are put in separate microservices, since they get used as components of other applications. Hence, the microservices approach. If there were startup dependencies across apps, the CI/CD pipeline that pushes the apps to the cloud would need to manage these dependencies. The dependencies in ref-app are simply the obvious ones, read-on.
While it's true that coupling of microservices is not in the design. There are some obvious reasons this might happen. Language and function. If you have a "back-end" microservice written in Java used by a "front-end" UI microservice written in Javascript on NodeJS then these are pushed as two separate apps. Theoretically the UI won't work too well without the back-end, but there is a plan to actually make that happen with some canned JSON. Still there is some logical coupling there.
The nice things you get from microservices is that they might need to scale differently and cloud foundry makes that quite easy with the 'cf scale' command. They might be used by multiple other microservices, hence creating new scale requirements. So, thinking about what needs to scale and also the release cycle of the functionality helps in deciding what comprises a microservice.
As for ordering, for example, the Google Maps api might be required by your application so it could be said that it should be launched first and your application second. But in reality, your application should take in to account that the maps api might be down. Your goal should be that your app behaves well when a dependent microservice is not available.
The 'apps' of the 'application' know about each due to their name and the URL that the cloud gives it. There are actually many copies of the reference app running in various clouds and spaces. They are prefaced with things like Dev or QA or Integration, etc. Could we get the Dev front end talking to the QA back-end microservice, sure, it's just a URL.
In addition to the aforementioned, etcd (which I haven't tried yet), you can also create a CUPS service 'definition'. This is also a set of key/value pairs. Which you can tie to the Space (dev/qa/stage/prod) and bind them via the manifest. This way you get the props from the environment.
If micro-services do need to talk to each other, generally its via REST as you have noticed.However microservice purists may be against such dependencies. That apart, service discovery is enabled by publishing available endpoints on to a service registry - etcd in case of CloudFoundry. Once endpoint is registered, various instances of a given service can register themselves to the registry using a POST operation. Client will need to know only about the published end point and not the individual service instance's end point. This is self-registration. Client will either communicate to a load balancer such as ELB, which looks up service registry or client should be aware of the service registry.
For (2), there should not be such a hard dependency between micro-services as per micro-service definition, if one is designing such a coupled set of services that indicates some imminent issues such as orchestrating and synchronizing. If such dependencies do emerge, you will have rely on service registries, health-checks and circuit-breakers for fall-back.
When running a Service Fabric cluster, it would make sense to have multiple applications running in it, but those applications might not be dependant on each other in any way. For example, I can have a CustomerApp in there, and a WikiApp.
Now from a security standpoint, it would be great if the WikiApp could be isolated from the CustomerApp, as a Wiki clearly should not be able to connect to services from an App that is holding customer data. I could put authentication into the services of the CustomerApp itself to allow only calls from authenticated services, but in addition, it would be even better if the WikiApp would not even be able to connect or see the other App and not able to resolve an endpoint adress from the naming service.
So is there a way to really isolate applications from each other in Service Fabric with a platform feature? I could not find anything about it in the documentation, and I also doubt it's possible the way Service Fabric works, but it would be very useful.
And to be clear, I'm really talking about isolating applications (ApplicationTypes) from each other, not services within a single application.
There are some levels of isolation built in:
Application instances have process-level isolation, in that each application instance runs in its own process.
Node isolation is possible, using placement constraints, to "isolate" services from each other by constraining them to run on different nodes.
Container support will be available in the future, where applications and services can run inside containers for further environment and resource isolation.
Services can run under unique user accounts, which you can use to perform authentication yourself at the application level.
But unfortunately there is no fine-grained role-based access mechanism built in to the platform today. So, for example, system-wide operations like running queries to get a list of applications or services or resolving endpoints using the naming service doesn't have any role-based access built in.
I'm trying to upgrade a Service Fabric application with a mix of stateful and stateless actors. I did some refactoring and so removed some actors I didn't need any more. Now, when I try to upgrade the application, I get the following error:
Services must be explicitly deleted before removing their Service Types.
After thinking about it a little bit, I think I understand the trouble that could come from removed services and upgrades, but then what's the correct way to do this?
You need to remove the service instances before you can upgrade to a version that doesn't contain the removed service package. Either:
In SF Explorer, navigate to the service and click Actions > Delete Service
In PowerShell:
Connect-ServiceFabricCluster
Remove-ServiceFabricService -ServiceName fabric:/MyApp/MyService
DO BE CAREFUL - If you're deleting a stateful service you'll lose all its data. Always be sure to have a periodic backup of production data.
I have a stateless ASP.NET Core (RC1) service running in my Azure Service Fabric cluster. It has the following manifest:
<ServiceManifest Name="MyServicePkg" Version="1.0.2" ...>
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyServiceType" />
</ServiceTypes>
...
</ServiceManifest>
My cluster is configured with placement properties. I have 5 servers with "nodeType=Backend" and 3 servers with "nodeType=Frontend".
I would like to upgrade my Service and specify that it may only be placed on "Backend" nodes. This is my updated manifest:
<ServiceManifest Name="MyServicePkg" Version="1.0.3" ...>
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyServiceType">
<PlacementConstraints>(nodeType==Backend)</PlacementConstraints>
</StatelessServiceType>
</ServiceTypes>
...
</ServiceManifest>
However, if I now execute the upgrade, I get the following error:
Start-ServiceFabricApplicationUpgrade : Default service descriptions
must not be modified as part of upgrade. Modified default service:
fabric:/MyApp/MyService
Why isn't it possible to change the constraints with an upgrade?
Would I have to delete and re-create the service? This would seem extremely problematic to me because it would result in downtime and data loss for stateful services.
So the issue here is actually with the DefaultService part of the ApplicationManifest. When services are created as part of the DefaultService, there are things you can't change about it afterwards. You might be able to change it through the ServiceFabric explorer, but I'm not sure.
One recommendation would be to keep the DefaultServices empty in the ApplicationManifest, and instead create your services manually. With manual I mean either through powershell, code or the ServiceFabric Explorer.
That gives you more flexibility about changing parts of the service afterwards. When it's done that way, you I know you have the possibility to change things like placement constraints after the service is running.
To create Services with PowerShell you can use the New-ServiceFabricService command.
To create it from code, you can use FabricClient to do it. A sample of that can be found here: Azure Service Fabric Multi-Tenancy
There's actually a fairly easy way to do this without having to write a bunch of code to manually define the application on the fabric cluster.
While you can declare the placement constraints in the service manifest, you can also declare them in the application manifest. Anything declared in the application manifest will override what's in the service manifest. And with the setting in the application manifest, you can then use parameters to alter the values based on the parameter file you want to a specific deployment.
I've just written up a blog post that discusses this approach in greater detail. I hope you find it useful. :)