We currently have an application that is based on an in-house developed workflow engine with YAML based DSL. We are looking to move parts of it to Java.
I have discovered a number of java solutions like Intalio, JBPM, Drools Expert, Drools Flow etc.
They appear to be aimed at businesses where the business analyst creates the workflows using a graphical editor and submits them to the workflow engine. They seem geared towards ease of use for non-technical people rather than for developers with a focus on human interaction.
The workflows tend to look like.
Discover-a-file -\
-> join -> process-file -> move-file -> register-file
Discover-some-metadata -/
If any step fails we need to retry it X times. We also need to be able to stop the system and be able to restart it and have it continue from where it was (durable).
Some of our workflows can be defined by a set of goals we need to achieve so Jess's backwards rule chaining sounds interesting but it is not open source.
It might be that what we are after is a Finite State Machine engine or just an Enterprise Service Bus and do everything as JMS queues.
Is there a good open source workflow engine that is both standards-based but also geared towards developers. We don't particular want to use a graphical workflow designer or write reams of XML and it should ideally be in Java or language agnostic (makes REST/Soap calls to external services).
Thanks,
Tom
Both Activiti and Bonita are open source and standard based (BPMN2). See for example this blog post.
Ruote is not standard based but seems close to your DSL approach and runs on a JVM thanks to JRuby.
Intaloi an open source BPM engine it offers a BPMN-support Designer and a BPEL engine. it's written in Java.
Camunda BPM is a developer-friendly Open Source workflow engine that is based on the open standards BPMN 2.0, DMN 1.1 and CMMN 1.1.
While it does come with a comfortable graphical workflow designer it also ships with a fluent API to build workflows programmatically. Camunda is written in Java, but can also be invoked from other languages via its REST API and it can make REST/Soap calls to external services.
jBPM 5 (open source, ASL, BPMN2) is just released and it's the best of Drools Flow and jBPM 4. It's lightweight but it can also integrate deeply with the Drools rule engine to make decisions.
For anyone looking for Python based enterprise grade solution.
Zengine, is GPL3 BPMN workflow based framework with Tornado, Rabbit AMQP, advanced permissions, extensible scaffolding features and more.
Built on top of following major components;
SpiffWorkflow: Powerful workflow engine with BPMN 2.0 support.
Tornado: Tornado is a Python web framework and asynchronous networking library.
Pyoko: Django esque ORM for Riak KV store.
RabbitMQ: Fast, ultrasharp AMQP server written with legendary Erlang lang.
Related
How is temporal.io related to cadenceworkflow.io? What should be used if starting a new project depending on the cadence workflow service?
Disclaimer: I'm the original co-founder and tech lead of the Cadence project and currently co-founder/CEO of the Temporal Technologies.
temporal.io is the fork of the Cadence project by the original founders and tech leads of the Cadence project Maxim Fateev and Samar Abbas. The fork is fully open source under the same MIT (with some SDKs under Apache 2.0) license as Cadence. We started Temporal Technologies and received VC funding as we believe that the programming model that we pioneered through AWS Simple Workflow, Durable Task Framework and the Cadence project has potential which goes far beyond a single company. Having a commercial entity to drive the project forward is essential for the longevity of the project.
The temporal.io fork has all the features of Cadence as it constantly merges from it. It also implemented multiple new features.
Here are some of the technical differences between Cadence and Temporal as of initial release of the Temporal fork.
All thrift structures are replaced by protobuf ones
All public APIs of Cadence rely on Thrift. Thrift object are also stored in DB in serialized form.
Temporal converted all these structures to Protocol Buffers. This includes objects stored in the DB.
Communication protocol switched from TChannel to gRPC
Cadence relies on TChannel which was TCP based multiplexing protocol which was developed at Uber. TChannel has a lot of limitations like not supporting any security and having a very limited number of language bindings. It is essentially deprecated even at Uber.
Temporal uses gRPC for all interprocess communication.
TLS Support
Cadence doesn't support any communication security as it is a limitation of TChannel.
Temporal has support for mutual TLS and is going to support more advanced authentication and authorization features in the future.
Simplified configuration
Temporal has reworked the service configuration. Some of the most confusing parts of it are removed. For example, the need to configure membership seeds is eliminated. In temporal each host upon startup registers itself with the database and uses the list from the database as the seed list.
Release pipelines
Cadence doesn't test any publicly released artifacts including docker images as its internal release pipeline is ensuring the quality of the internally built artifacts only. It also doesn't perform any release testing for dependencies that are not used within Uber. For example, MySQL integration is not tested beyond rather incomplete unit tests. The same applies to the CLI and other components.
Temporal is making heavy investment into the release process. All the artifacts including a full supported matrix of dependencies are going to be subjected through a full release pipeline which is going to include multi-day stress runs.
The other important part of the release process is the ability to generate patches for production issues. The ability to ensure quality of such patches and produce all the necessary artifacts in a timely manner is important for anyone running Temporal in production.
Payload Metadata
Cadence stores activity inputs and outputs and other payloads as binary blobs without any associated metadata.
Temporal allows associating metadata with every payload. It enables features like dynamically pluggable serialization mechanisms, seamless compression, and encryption.
Failure Propagation
In Cadence activity and workflow failures are modeled as a single binary payload and a string reason field. Only Java client supports chaining exceptions across workflow and activity boundaries. But this chaining relies on fragile GSON serialization and doesn't work with other languages.
Temporal activity and workflow failures are modeled as protobufs and can be chained across components implemented in different SDKs. For example, a single failure trace can contain a chain that is caused by an exception that originates in activity written in Python, propagated through Go child workflow up to Java workflow, and later to the client.
Go SDK
Temporal implemented the following improvements over Cadence Go client:
Protobuf & gRPC
No global registration of activity and workflow types
Ability to register activity structure instance with the worker. It greatly simplifies passing external dependencies to the activities.
Workflow and activity interceptors which allow implementing features like configuring timeouts through external config files.
Activity and workflow type names do not include package names. This makes code refactoring without breaking changes much simpler.
Most of the timeouts which were required by Cadence are optional now.
workflow.Await method
Java SDK
Temporal implemented the following improvements over Cadence Java client:
Workflow and activity annotations to allow activity and workflow implementation objects to implement non-workflow and activity interfaces. This is important to play nice with AOP frameworks like Spring.
Polymorphic workflow and activity interfaces. This allows having a common interface among multiple activity and workflow types.
Dynamic registration of signal and query handlers.
Workflow and activity interceptors which allow implementing features like configuring timeouts through external config files.
Activity and workflow type name generation improved
SDKs not supported by Cadence
Typescript SDK, Python SDK, PHP SDK
SDKs under active development
.NET SDK, Ruby SDK
Temporal Cloud
Temporal Technologies monetizes the project by providing a hosted version of the Temporal service. There are dozens of companies (including SNAP) already using it in production.
Other
We have a lot of other features and client SDKs for other languages planned. You can find us at Temporal Community Forum.
Overview
Using iWF will let you switch between Cadence & Temporal easily.
In addition, iWF will provides a nice abstraction on top of the both and makes your life a lot better.
The fact is both Cadence & Temporal are under active development. You can see they have some different focuses if looking at their road maps. The two projects share the same vision to let everyone rethink about programming models of long-running business.
Tasks across domain+clusters
If you have multiple Cadence clusters,
this allows starthing childWF across different clusters and domains.
Support Both Thrift&gRPC
gRPC support is completely done on the server side. Internal traffic is all using gRPC and we are working on letting users migrating from Thrift to gRPC.
Authorization
The permission is based on domain but can be extended. Different from Temporal, the permission policy can be stored within Cadence domain data storage so that you don't have to build another service/storage to manage them.
Note that the whole proposal is developed by community member.
Workflow Shadower
Workflow Shadower is built on top of Workflow Replayer to address this problem. The basic idea of shadowing is: scan workflows based on the filters you defined, fetch history for each workflow in the scan result from Cadence server and run the replay test. It can be run either as a test to serve local development purpose or as a workflow in your worker to continuously replay production workflows.
Graceful domain failover
This allows XDC(multiple clusters) mode to reduce the pain of rerun some tasks during failover.
NoSQL plugin model
This allows implementing different NoSQL persistence in a minimum way. By the time writing this post, Temporal haven't started working on it.
MongoDB support
On top of the NoSQL interfaces, MongoDB support is WIP.
Using multiple SQL instances as sharded SQL
This allows user to have a Cadence cluster with a much larger scale. (then using XDC to add more DB instances)
Configuration Storage for Dynamic config
This enables us changing the dynamic configuration(like for ratelimiting) without making any deployment. Just a CLI command can control the behavior of the system.
It's in experiment and still WIP for production ready.
Workflow notification
A WIP eco system project to allow getting notification from Cadence. This is the benefits of Cadence using Kafka to deliver visibility messages. Temporal doesn't use Kafka which will be super difficult to support this feature.
Periodical Healthchecker(Canary) and Benchmark tool and benchmark setup docs
More Documentation
Seamless Cluster Migration guidance
Dashboard/Monitoring
...
...
Other small improvements that Temporal is missing
TerminateIfRunning IDReusePolicy
All domain API forwarding policy
Better & cleaner XDC configuration
Tooling to deserialize database blob data
...
...
I'm from the Cadence team at Uber, and I wanted to let you know that Cadence continues to be developed actively by our team. Below is a section of the update that we shared with the Cadence community recently:
We want to reinforce that Uber's Cadence team is committed to the
growth and open source development of the Cadence project. Today,
Cadence powers 100+ different use cases within Uber and that number
grows quickly. Collectively, there are 50M+ ongoing executions at any
moment on average and our customers finish 3B+ executions per month.
Outside of Uber, we also know that many engineering teams at various
companies have already adopted Cadence for their business-critical
workflows. We are excited to continue evolving Cadence as an
open-source project in a backward-compatible way with an increased
focus on reliability, scalability, and maintainability in the near
term.
It's probably too early to compare Cadence and Temporal. Still, I have a few ideas around how we can systematically shed light on Cadence's roadmap to ensure all the necessary information is out there to enable such comparisons going forward. I'll update this post with links when we create a page with information about the roadmap.
In the meantime, please let me know if you need further information about Cadence that would be helpful in this context.
Temporal.io is a company that has forked cadence project and are now building on top of it - naming it temporal.
It is founded by the authors of cadence.
I would suggest using temporal.io as it is under active development
We are trying to choose an approach we should follow in integrating on-promise Dynamic CRM with other companies and third-party software and systems, we use in our company.
Concerning this msdn page: https://msdn.microsoft.com/en-us/library/dn932127.aspx we have some options. But my team members want to implement integration through service calls to external systems inside plugins.
I am talking about Data Integration and Data Synchronization for a total solution that Dynamics CRM is a part of it. Considering the limitations in plugin and the options written in the mentioned msdn page, I think bringing the integration code and logic out of the CRM stuffs and encapsulate it in into a separate connector solution would be a better solution. The connector of course
should support real-time and offline or on-demand data synchronization.
I want to know what are the disadvantages/advantages in plugins approach in integration with external systems, and what are advantages/disadvantages of using a connector solution?
Thanks
Implementing integration to third party system within plugins can be a good approach. You should be aware of how plugins work and how the result of your integration affects the success or failure of the database action that triggers the plugin. Some important items to consider:
Is the database action dependent upon a result from the integration. For instance if your integration is verifying/modifying your data, you may want the result of the integration to affect what is written to the database. Such a plugin would probably be registered on a pre plugin.
Should the database action go through if the integration fails? Many integration are done in a post plugin outside of the transaction so that if the integration fails it does not cause the database action to fail.
This really sounds like something you might want to solve using tooling outside of CRM. For example Scribe, SSIS (with Kingswaysoft) or some ESB type application.
Especially the first two provide no-code solutions and would probably have a lower TCO than writing your own plugins.
We are evaluating Activiti as a process engine to replace our existing home grown work flow engine. We are quite impressed by the capabilities of Activiti especially related to multi tenancy and REST WS.
However, one of the biggest challenge (and probably blocker) to adopt Activiti would be - How we can run or migrate our existing work flow definitions.
As I mentioned earlier, our work flow solution is a home grown one and doesn't adheres to BPMN specifications. There are thousands of templates out there. We can't simply ask our customers to redefine their templates using Activiti. These definitions are stored in proprietary XML format.
Looking at the level of customization in the templates, it would be very difficult to migrate these definitions to BPMN format.
So, does Activiti provides any hooks to run such custom templates. Alternatively, please share your thoughts about migrating the templates from proprietary format to BPMN format.
I suppose such scenario would be common and other people would have faced the same.
I know I am being very vague with this query but at this stage I don't have specific problems that I can discuss.
One option is implement your own proprietary XML parser and parser handlers. Look at org.activiti.engine.impl.bpmn.parser.BpmnParse and org.activiti.engine.impl.bpmn.parser.handler.AbstractBpmnParseHandler and its descendants.
We did it and worked fine.
Can Drools be used with non-Java applications? Most examples that I've seen are in conjunction with Drools interfacing with Java applications.
EDIT: Looks like http://docs.jboss.org/drools/release/5.5.0.Final/droolsjbpm-integration-docs/html_single/ describes the Drools API on techniques on getting information in and out of Drools.
I would suggest to expose the functionality that you need from Drools via REST or SOAP and interact with it from your application using that. I wouldn't spend time checking the Drools .NET port is old and out of dated.
A JS implementation is a good idea.. but just for specific cases.. you don't want your whole business rules running on the client side.. (for a web app of course).
HTH
Well, it doesn't have to be Java, but it does have to be a JVM language (I've used Groovy in the past with Drools). Even then, there's nothing to prevent you from calling a REST service, for example, when a rule fires, and interacting with another system that way.
That said, there are other rules engines in other languages. I believe there is a .NET Drools port, and a Javascript implementation.
I am looking for a robust REST framework to eliminate all that boilerplate code with starting up a new REST-only web service (mobile clients). Is there a framework that already has this built-in where I could, for example, simply build the domain models and run with it? I would like to see:
Authentication & User Model
Logging
Basic CRUD
Permissions (for model access)
Scalability
It seems every web service at a minimum needs the above capabilities. Somebody, somewhere must have written a good re-usable framework with the above capabilities. Any ideas? I would prefer Node.js, Java or even hosting with a PaaS service provider that offers these features.
Spring 3 MVC provides a very nice and simple annotation based framework for REST.
See http://blog.springsource.org/2009/03/08/rest-in-spring-3-mvc/ it can be deployed on any java web server like Jetty or Tomcat.
A framework like XAP provides a combined solution of Spring and Jetty plus it's built for dynamic scaling.
See http://www.gigaspaces.com/xap.
Last if you want to easily on board this solution on any cloud CloudifySource provides an open source project which includes XAP capabilities and PaaS.
See http://www.cloudifysource.org
I use Symfony 1.4 for this. It is an PHP framework. It generates most of what you need for free. The database stuff is also quite easy as the Symfony uses ORM libraries (you can choose but I can recommend Doctrine: http://www.doctrine-project.org/).
For example the whole backend site(admin) generating is a matter of running one command. They have a great e-book fro free. More info here:http://www.symfony-project.org/.
There is also Symfony 2.X (http://symfony.com/), which have a lot of new features (e.g. new Doctrine 2.0). Especially with the bundle (plugin) https://github.com/FriendsOfSymfony/FOSRestBundle is the RESTful service quite easy.