we are evaluating JBPM and would like to understand if following is support is available with community edition
Model validation : does it validate against the whole process and identify issues such cycle loops or dead locks?
Adaptive/ Event based dynamicity support: As events occur, a person or system chooses the appropriate activity. The resulting steps can vary depending on the event and choices made by the worker. Activities are not directly linked to one another.
Multi lingual support: which langauges supported
Workflow load predictions based on historical data?
Process related
Process simulation Support
Process Optimization Support
Dynamic Process Support
Supporting failure recovery : After failure, once restarted, the engine reprocesses all pending events, after having recovered its last stable internal state.
HIGH Availability based clustering Production deployment support
Availability of Production Support for community edition deployment (on chargeable/ subscription)
Availability of Commercial Remote & On-Site Training. (on chargeable/ subscription)
Please find the response inline (AFAIK).
Model validation: does it validate against the whole process and identify issues such cycle loops or dead locks?
There is an option to validate for every type of asset in jBPM which covers various scenarios and provides warnings and error messages.
Adaptive/ Event based dynamicity support: As events occur, a person or system chooses the appropriate activity. The resulting steps can vary depending on the event and choices made by the worker. Activities are not directly linked to one another.
Event-based gateways allow reacting to possible Events and
Multilingual support: which langauges supported
Not sure.
Workflow load predictions based on historical data?
Not Sure
Process related Process simulation Support Process Optimization Support Dynamic Process Support
Supporting failure recovery : After failure, once restarted, the engine reprocesses all pending events, after having recovered its last stable internal state.
Yes, jBPM persists internal states and runtime details of the process in DB (Default is H2 DB), refer Documentation to learn more about it.
HIGH Availability based clustering Production deployment support
Refer Link for configuring business-central high availability.
Availability of Production Support for community edition deployment (on chargeable/ subscription)
Subscription-based support is available for Red Hat Process Automation Manager not for community edition. You can check with RedHat sales for more details.
Availability of Commercial Remote & On-Site Training. (on chargeable/ subscription)
Red Hat does provide various training courses for the RedHat process automation manager.
Related
Cadence is a fault tolerant stateful code platform. How does cadence handle fault in various failure condition?
There are al kinds of failures in distributed systems and Cadence provides various options to them.
Here is the list from myself. It may not be complete. But I will try add more if I can think of.
activity
Activity failure and retry. See https://cadenceworkflow.io/docs/concepts/activities/#timeouts
Also note that long running activity can recover from checkpoints via “heartbeat “
workflow
By design of event sourcing models, a workflow can recover from any point left when a worker crashed. See https://cadenceworkflow.io/docs/concepts/workflows/#state-recovery-and-determinism
Workflow can also have retry policy like activity to retry on failure automatically https://cadenceworkflow.io/docs/concepts/workflows/#workflow-retries
On certain scenarios the failure is caused by bad code change which leads to wrong states. Cadence provides “reset” tool to reset workflow to any point of time.
See https://cadenceworkflow.io/docs/cli/#reset-and-restart
On top of reset, Cadence also allows you to reset by deployment. This is useful to reset a big number of workflow(eg millions of).
Cadence server cluster
Both activity and workflow workers are stateless.
Cadence server is a highly available and scalable service provides the durability.
The durability is from underlying design and persistence storage ( by either Cassandra, MySQL or Postgres)
In a single cluster setup, Cadence service is running with different independent shards. The whole cluster consists of different hosts. Any failed host can be replaced by another.
Cadence provides Cross data center replication to provide much higher availability https://cadenceworkflow.io/docs/concepts/cross-dc-replication/#global-domains-architecture
I want to create a distributed system that can support around 10,000 different types of jobs. One single machine can host only 500 such jobs, as each job needs some data to be pre-loaded into memory, which can't be kept in a cache. Each job must have redundancy for availability.
I had explored open-source libraries like zookeeper, hadoop, but none solves my problem.
The easiest solution that I can think of, is to maintain a map of job type, with its hosted machine. But how can I support dynamic allocation of job type on my fleet? How to handle machine failures, to make sure that each job type must be available on atleast 1 machine, at any point of time.
Based on the answers that you mentioned in the comments, I propose you to go for a MQ-based (Message Queue) architecture. What I propose in this answer is to:
Get the input from users and push them into a distributed message queue. It means that you should set up a message queue (Such as ActiveMQ or RabbitMQ) on several servers. This MQ technology, helps you to replicate the input requests for fault tolerance issues. It also provides a full end-to-end asynchronous system.
After preparing this MQ layer, you can setup you computing servers layers. This means that some computing servers (~20 servers in your case) will read the requests from the message queue and start a job based on the request. Because this MQ is distributed, you can make sure that a good level of load balancing can happen in your computing servers. In addition, each server is capable of running as much as jobs that you want (~500 in your case) based on the requests that it reads from the MQ.
Regarding the failures, the computing servers may only pop from the MQ, if and only if the job is completed. If one server is crashing, the job is still in the MQ and another server can work on it. If the job is saving some state somewhere or updates something, you should manage its duplicate run then.
The good point about this approach is that it is very salable. It means that if in future you have more jobs to handle, by adding a computing server and connecting it to the MQ, you can process more requests on the servers without any change to the system. In addition, some nice features in the MQ like priority-based queuing, helps you to prioritize the requests and process them based on the job type.
p.s. Your Q does not provide any details about the type and parameters of the system. This is a draft solution that I can propose. If you provide more details, maybe the community can help you more.
I am developing a customer account system for a chain of recycling centers in the Northwest US. One of our key features is that our customers can set up accounts that are credited with their bottle deposit refunds, instead of always disbursing cash. Customers can also drop off bags of recyclables that are processed on-site and credited. Each center runs near capacity and can physically process cans and bottles when offline, so we don't have a lot of leeway for IT infrastructure to shut down everything when the Internet goes out.
Basically, I've been asked to develop a customer account system that will allow credits from a retail center to be posted to accounts, even if telecommunications with our central server breaks down for a period of hours. This will allow the center to keep processing and crediting customers when the pipes get clogged. Certain transactions, like withdraws, do NOT need to occur in this situation, since we can't accurately get the customer's current balance.
We are a 100% Windows shop, and the IT manager and network admin don't want to get near anything *nix. Each retail center has an on-premise dedicated Windows Server, so that seems like a logical place to start.
I'm a huge fan of ServiceStack, and the REST-ful message-based paradigm seems like might work. I'd create a "Credit" message and send it to the local server. A message broker there would log the request and attempt to forward that message to the central server where it is processed. In case the central server were down, I would rely on the MQ's reliable messaging protocol to hold on to it until telecommunications are restored. The overall anticipated volume is 100s to low 1,000s of messages out of each center, so low by modern computing terms.
The Redis MQ Client / Server for ServiceStack looks interesting, but since the Windows Redis server is explicitly labeled "prototype" and "not production quality", there is a 0% chance of being able to leverage it.
So, ultimately the questions are:
Is a reliable messaging system the right type of solution for this problem? Are there other approaches I should consider?
Are there alternatives to Redis that play well with ServiceStack? Is there a "production quality" NoSQL server replacement I can use on Windows?
I've looked briefly at RabbitMQ. Might that be an option? My Googling doesn't show any active integration between it and ServiceStack, so I'm leery of writing something from the ground up.
Ideally the overhead of my solution is low enough we can perform a synchronous update and return a "current balance" receipt to a customer if everything is working well. Is this a realistic?
A production solution for running Redis on windows is to run redis-server inside a Linux VM on windows with Vagrant.
There is current a feature request to add more MQ Options to ServiceStack. Rabbit MQ is expected to be the next MQ adapter to be supported in future.
As a follow-up, MS Open Tech has released a "production-ready" native implementation of Redis 2.8.9. GitHub link.
We need to send an XML messages between a point of sale system and a java webservice (outside of our network). the messages contain very sensitive data. The messaging has to be secure and transactional and highly available (24/7) with failover. The solution requires the developement of a broker that does the following:
Poll messages from the POS of system (3 types of messages)
do some transformation to the messages
forward part of the message to the java webservice
store part of the message in a database
notify the POS system of the result
Based on these somewhat simplified requirements, do you believe that Biztalk would be overkill? would MSMQ/WCF do the trick here?
Thank you for your help
Amine
IMO if you have the ability to receive and deliver messages asynchronously, then MSMQ (or other Message Oriented Middleware) would be an obvious choice for reliable, transactional transport, irrespective of the rest of the solution. MSMQ's journalling can also be used for audit and debugging purposes (but you will need a strategy for archiving the journal).
For the Polling, Routing, Mapping / Broker and Auditing requirements you then have the choice of BizTalk, other ESB and EAI products, or a DIY solution.
As you've suggested, it is difficult to justify the cost and learning curve of BizTalk on a single message exchange scenario such as this - you could probably knock up a .NET Windows Service (e.g. using WCF, Workflow Foundation, Transaction Scopes, some XSLT for mapping and a data access layer) in a few days.
However, if this isn't a one-off integration scenario and the need for additional integration arises (more applications to integrate, more services, additional listeners, different communications technologies etc), then it would be advisable for your company to take a long term view on EAI and ESB technologies. IMO the main challenge in integration isn't the initial development work, but is instead the ongoing operational management requirements - e.g. security, auditing, failover, monitoring, handling of bad messages and other exceptions - where products such as BizTalk are really worth the outlay.
Do you want to and have the bandwidth to develop, monitor, and maintain your own custom solution? If you don't mind doing that, then going the route of a custom .net-based, MSMQ/WCF solution might work well.
BizTalk will also cover all of the requirements you have listed. There is a learning curve but it is certainly not insurmountable. The initial ramp-up may be lengthier than would a custom-code solution, but there are considerable benefits, particularly the benefit of having all your requirements reliably met:
secure
transactional
reliable (messages aren't lost)
highly available (24/7)
failover
adapter architecture (includes polling adapters)
transformations
working with external web services
returning correlated responses back to the source system (i.e., orchestrating the end-to-end process)
use a broker (you specifically listed this, and BizTalk is a broker; custom MSMQ and WCF means using no broker)
If BizTalk needs to poll the POS system, then you do not need to worry about using MSMQ. BizTalk can handle transferring messages reliably (they're persisted to SQL Server, while MSMQ persists messages to disk).
Note too that the only way to make MSMQ highly available is to cluster it. So either way you'll need to cluster something.
A BizTalk solution will be easier to maintain over time, particularly if you just want to update your transformations. With versioning you can do so in a way that doesn't require downtime. It'll be tough to update a custom solution without downtime.
Some people have had difficulty in the past with monitoring BizTalk for failed messages, but I have found it to be easier, especially with a tool like SCOM or BizTalk 360, than trying to monitor message queues, which often requires even more custom work to monitor. Just make sure to include monitoring in your cost estimates for the life of your solution.
If you do need auditing, then BizTalk also has you covered. MSMQ Journaling will keep a copy of each message for you, but without significant transaction details and with no out-of-the-box way to search through or archive the data.
Building your own .NET client code to work with a Java web service will likely take a good bit of work regardless of which way you go. With BizTalk that means running a wizard against the endpoint or against the WSDL. With WCF it means doing everything by hand or with the assistance of the svcutil tool.
You should go with MSMQ transporting either way.
If you use MSMQ from .NET you should know its limitation: 4 MB on a message size.
BizTalk on the other hand has MSMQ adapter which overcomes this limitation (if a second BizTalk server listen on the other side of the channel).On top of that BizTalk gives you features like: easy configurable message tracking, visual transformation maps. It can be set up in cluster too (Ent. version only).
But the question is can you (or do you want) afford biztalk licenses and hardware for it servers (it's slower then custom .net solution).
I want client machines on internet who subscribe to my server to donate their idle cpu cycles.(Like SETI#Home)
They would take jobs(work-units) from server to process, and send back results to the server. (This is the most simple description). The framework i need should allow me to define a job/task. Rest of things like communication, job execution/tracking, client binaries update etc. should be managed by framework.
I evaluated Alchemi.NET a bit, but its not actively maintained, seems half-baked.
BOINC has API in C, but i want a .NET or JAVA framework.
I am looking at Manjrasoft's ANEKA , but it seems to work only for LAN clouds.
There must be some such frameworks available. I need expert recommendations!
I'm hardly an expert but I do have a little experience with distributed computing using MPI (with C). What you're talking about does not sound like grid computing, rather a master/slave system. That is your server is the master and it directs all the clients (slaves).
I know very little about .net programming so I'll speak in general.
There are a lot of web frame works out there and probably most have the facilities you'll need. That is your client will be able to upload files with the content they have gathered (or they could just use http get/post), because you don't have to worry about UI issues you can handle everything probably through one action (assuming an action based web framework). Then the server can return a response via JSON or XML which the client can use to take further direction on. JSON is the right choice if the system is very simple and probably a good choice for prototyping.
For the application upgrade I would consider this a separate issue (although it should be a simple matter for the server to return this to the client).
BOINC is the framework that most naturally meets your volunteer computing requirements, and it's stable and highly scalable -- I'd make sure you've considered it completely before ruling it out.
If you need to deliver something to a short deadline then I'd consider working up a simple supervisor (or scheduler) - worker pattern, just to get the application off the ground. The supervisor would be responsible for chunking up the data and making it available over http. Workers (your client app) would connect to a supervisor server; download a chunk of work; complete the chunk; and upload the results to the supervisor.
The main trick is to get the state machine thrashed out properly, so that you can accurately track what state each work chunk is in. I'd have the supervisor persist state in a database in the background.
Does your first release need to be internal, or is it for public consumption?
While that's working I'd get started on looking at getting up to speed with BOINC and planning a migration.
My recommendation
Work dist:
Have a receiver of requests, that places messages in a message queue, like rabbit mq
Have a host of workers, listening to the same queue, taking work from it and acking it when done.
When done, send a message on another queue, containing an URI to a known location, such as your network drive. The target is your parsed data.
The receiver listens to these "completed" messages. Fetches the data from the URI.
Done.
RabbitMQ comes with great CLR APIs.
The same reasoning works well with Microsoft Azure and their AbbFabric Queue. A plus is that is scales extremely well.
Hot Versioning
http://topshelf-project.com/
It gives a folder where you can drop binaries, which are then run. It manages versioning of these as well as running them as windows services.
Deployment
You can deploy the binaries with robocopy/xcopy and "net use Q: pwd \server\share", "net delete Q:"
Continuous Integration
Teamcity
After working with MsBuild extensively, I would recommend scripting it with psake and running the build with PowerShell. If you get advanced with PowerShell you also have WinRM available to you from your build scripts, which is really smooth.
Use the git/subversion commit number as the 0.0.0.x, x in the previous version number, and you will have automatic versioning that is "shared" across "Debug"/"Production" builds.
The Azure way
Work dist:
Same as above but with AppFabric Queue instead of RabbitMQ.
Hot Versioning
By swapping "Staging" and "Production" instances around, you avoid the downtime.
Deployment
You can either tap into the Azure Tools for Visual Studio's MsBuild tasks as can be read about here or you could use the PowerShell AzureSnapIns with a similar setup as above for Continuous Integration.
Continuous Integration
Same as above.
How about .net's ClickOnce installer to manage auto updated client binaries.
http://msdn.microsoft.com/en-us/library/t71a733d.aspx
I'm not sure of a "jobs framework" per-say, but Microsoft's Sync framework to support rolling your own jobs syncing with clients?
http://msdn.microsoft.com/en-us/sync/default