I'm an application developer mainly develop and maintain enterprise application, like ERP, HCM system. After being in the field for many years, I started feeling that the way business system are developed is not quite right. After years of maintenance and enhancement by hundreds of developers, the system keeps getting bigger and bigger, more and more complex. At the end, it just impossible to do big changes in the system, because the logics are all tangled together like Italian noodles. Developers so afraid of causing severe customer issues.
Recently I find Flow based programming paradigm invented by J. Paul Morrison and I find it really interesting. I approve very much the idea of doing application development by drawing diagram visually. As we all know to develop business system we start with drawing business flow diagram. Why can't business flow diagram just be the system itself??
Naturally, I tried to find FBP implementations, and nifi is the one that the FBP inventor recommends. I haven't dig very deep into nifi.
Just after watching some introduction videos and documentation, I find most of the time, the nifi experts always talking about using nifi for iot system, real time streaming these kind of stuff. It seems that nifi is not related to business systems.
Looking forward to someone to clarify my doubts. Is nifi suitable for building business transactional systems?
Apache NiFi is definitely used for many "business logic" systems, especially taking on the role of handling extract/transform/load logic (ETL). While not strictly an ETL tool, NiFi can facilitate data routing and simple event processing in a number of scenarios. The "Powered By NiFi" page lists some public use cases of NiFi, and many are for "business systems" that do not relate to IoT.
sorry I didn't see your question before - your comments are interesting. I am surprised that you say that NiFi is the FBP software that I recommend - I do list it as a product that is closer to the "classical" FBP philosophy than what we call "FBP-like" or "FBP-inspired" systems, and I assume it is one of the few FBP products that are in the marketplace - unlike my work, which is all public domain. The terms "FBP-like" and "FBP-inspired" are actually thanks to Joe Witt, the developer of NiFi. I try to describe the difference between "classical" FBP and "FBP-like" in my article on https://jpaulm.github.io/fbp/noflo.html . With all due respect to Joe, I find NiFi a bit over-complex, although his data packets are immutable, which has certain advantages. For a complete suite that takes you from a diagram to actual running code, I would suggest you start with the FBP diagramming tool, https://github.com/jpaulm/drawfbp , generate a JavaFBP network, using https://github.com/jpaulm/javafbp , and run! Both of these tools, as well as others on https://github.com/jpaulm/ , are open source. My colleague, Bob Corrick, and I are working on a tutorial which you may find helpful: https://github.com/jpaulm/fbp-tutorial-filter-file .
Related
I'm currently writing a document during an internship regarding the WSO2 Enterprise Integrator.
Problem is, I'm not going to work with it and definitely not going to write a single line of code, yet my job would be creating some guidelines that dictate how services (both SOAP and REST) should be orchestrated through the ESB.
The WSO2 documentation isn't helping me a lot because it mostly delves into explaining how to set up the Integrator and what tools are avaiable, and online I can only find simple tutorials which I can't really understand anyway because, as I said, I'm not even working with the platform.
So basically I don't understand if there is a specific conduct to follow when orchestrating services like "add this specific function in the service, create a dependency to the ESB" or whatever.
Any form of suggestion or just directing me towards the right documentation would be super helpful
Under my experience, what you must do is a proof of concept performing service orchestration, now by definition one of the characteristics of any ESB is precisely to be an orchestrator. I work daily with WSO2 and allows it in simple ways (with the same mediators) or complex ways (extending or creating new components)
PS: Excuse me for my English.
Regards !!!
I want to dive into the whole diversity of tools which provide connection between programs over the network.
To clarify the question, I divide it on subquestions:
Why some groups of programs (or specific tools/frameworks/approaches with programming languages where this frameworks can be used) were popular in each period of time? (I expect description of problems which were solved, description of tools, why those tools are considered as best solution to those problems at that time, why some tools lost popularity)
What is the entire history of software communication over the network? (tools/approaches popularity precisely to decades)
What are the modern solutions to this problem?
I can distinguish only two significant approaches.
RPC, RMI and their implementations (I saw this, but it is about concrete problem and specific tools to solve this problem, I want to see the place of this problem in the whole picture of interconnection programs over the network. I heard about implementations: ONC RPC, XML-RPC, CORBA, DCOM, gRPC, but which are active now? which are reasonable to use? which are preferable and why? I want answers not to be opinion based, so I accept answers like "technology A better than technology B for problem X because ..." only if there is reliable research/statistics or facts). I heard that RPC and RMI were popular 10 years ago. Are they still?
Web services: REST, SOAP.
Am I miss something? Maybe there are some technologies which solve problem completely new way? Maybe there are technologies which can be treated as replacement to RPC(RMI) and Web Services? Can we replace RPC(RMI) by REST for any task? Can we replace RPC(RMI) by REST only for modern tasks? Should I separate technologies not as RPC and Web Services, but in some other manner?
As a partial answer, I can give you my feedback on the use of RabbitMQ.
As explain here, it provides a lot of different ways to use it :
RPC by implementing a "callback" queue
One to one, one to many routing strategy to propagate your events through your whole infrastructure and target the right destination.
It comes with the ability to persist messages to avoid loosing data when a crash appears but also with some plugins to increase possibilities (e.g x-delayed plugin)
This technologie written in Erlang is powerful and is a must try in term of communication between programs.
To your question „Am I missing something“: yes.
Very popular communication patterns are the so-called Event-Driven or Message-Driven protocols. This type of protocols are often used in distributed systems such web applications, microservices and IoT-Environments. The communication is complete asynchronously and allows building scalable and loosely coupled systems.
There are many different frameworks and methods for Event-Driven systems like WebSockets, WebHooks, Pub-Sub and Messaging-Librarys like AcitveMQ, OpenMQ, RabbitMQ, ZeroMQ and MQTT.
Hope this info helps for your research.
I spent short time studing Habanero and i found it good approach for making Enterprise Application in a really short period of time.
The pattern witch Habanero use is "Active Record" as it's developers say.
My questions are:
There any similar application like Habanero witch fully support Domain
Driven Design by determining aggregate roots, entities and value objects
Is it right decision to use such tools in big organizations
Does it worth training our team on such a tool
thank you
Framework support for Domain Driven Design is quite different from frameworks supporting data driven applications. Such framework should increase the productivity of developers that works with an ubiquitous language that evolves with the business and that is learned by a domain-expert.
They should not face concepts like aggregates, root, value objects because they are just modelling concepts, conceptual tools, but ways to ease the development process. Thus a framework exposing abstract classes or interfaces named AggregateRoot, Entity or ValueObject is fundamentally broken. It doesn't provides any real value to an application, just useless indirections.
However:
There are a few frameworks designed to support domain driven design, listed here. Moreover, I'm developing one by myself based on previous experiences that worked very well
It depends, obviosly. For example we used all of the Epic's modeling patterns with success.
We used some "home made" framewoks too, and some of them proved to really increase productivity. However, such frameworks (if useful) always have steep learning curves and it depends very much on how much reliable the software have to be and what are the developers skills.
It depends on the framework, on the complexity of the business (if you don't need a domain expert to understand it, you don't need DDD) and on the developers, too. I faced successful stories and huge failures with different frameworks in different contexts. I've also had a conference that faced the topic (you can see the slides here).
I'm in the process of deciding how to write an online auction application. I would like to provide real-time updates to the site users. My background is with LAMP (although, in my case, the 'P' would be more for Perl than PHP). I've considered ActiveMQ, but I'm wondering if there are better options.
My primary concerns are scalability and speed. It could have several simultaneous auctions taking place, with [hopefully] many users participating in each auction. Whatever solution that I decide on would have to accommodate such a scenario. Of course, this is all in theory so I have no idea how many concurrent users that I might have, but I'd like to have the means to support tens of thousands of users.
Another concern is ease of implementation. I've spent the past few days reading docs and tutorials and, so far, nothing has come across as anything less than a bit of a pain in the rear to deal with, which is actually what has led me here to seek some advice.
I was hoping to use a web framework, such as Codeigniter (PHP) or Catalyst (Perl), because I intend to pay a contractor or two to help with some of the bulk of the coding, and I like the idea of having a framework to somewhat enforce a design pattern. However, the more that I look into this, I'm just not seeing an obvious solution to 1) use a framework, and 2) provide real-time auction updates (other than Tornado, I guess - maybe I'm answering my own question. ;)).
So, with all that said, short of using polling (which I'm not really interested in doing), is there a way that I can accomplish these real-time updates using a language like Perl or PHP for my server-side code? I know that ActiveMQ supports STOMP, and I actually have this working on my local machine (using Jetty since it requires a servlet to publish/consume messages from client-side javascript), but is there a better option here?
I'm sorry that I don't have a more direct question, but after several days of looking at docs and tutorials, I'm more lost than ever!
Part of your problem is that your mixing a variety of concepts together. If I read things correctly you have a problem statement of:
I'm building an online auction site and would like to insure that my visitors have real-time updates of prices on the items they are viewing.
Now between the Browser and the Server you'll probably use a Comet style request pattern to handle communications, you could also look at socket.io as a backup pattern. This polling will require a server that is able to handle lots of simultaneous open connections, which Tornado is a good candidate (there are others, but given you asked in relationship to Tornado it's good).
Now that we've gone from 1000+ of Browsers to a handful of Tornado servers, you need a way to communicate between them. In the the last of publish/subscribe message patterns you have a few choices:
RabbitMQ (AMQP)
ZeroMQ
Redis Pub/Sub
All three a good choices, with their own pros/cons. Personally I've used Redis and Rabbit on different projects and just toyed with ZeroMQ. The message broker is a whole decision tree that is going to be based on what you have available.
Is there anyone who has used or looked into using Jitterbit as well as BizTalk? If so, what are some pros and cons of each, and which one did you go with as your final solution?
Specifically, I'm looking for SAP integration, but any input would be appreciated.
Like Rob I have not heard about JitterBit until reading your question (thanks!), I have, however, been working with BizTalk, almost exclusively, for the past 9 years; for that reason I wasn't sure I should be responding, but as Rob did, and nobody else has, I figured it's worth a couple of cents....
From the little reading I've done it seems to me that JitterBit, apart from being an open source, which has it's pros and cons, is trying to lower the entry barrier by offering a relative simple solution with the promise of rapid development and drag-n-drop approach "with no custom code".
I'll take their promise at face value, as I know nothing about it, although I have my doubts, so let's assume developing with JitterBit is really easy, there's one thing I can clearly state - developing with BizTalk isn't.
But, and that's a bit but in my view, developing with BizTalk is somewhat difficult not because Microsoft did a bad job at it, on the contrary - developing with BizTalk is somewhat difficult because Microsoft wanted to create a tool that could realistically allow enterprises to solve their BPM and integration needs well, and, in my experience, these problems are almost never simple, so Microsoft had built a server that has many capabilities, is very strong and very flexible, at the cost of complexity.
So, while any experienced technical sales guy can give you a demo of an integration scenario that is very simple, and is developed in a few minutes using a lot of drag and drop and configuration, even in BizTalk, but is this a realistic enterprise-level solution? was it a realistic scenario that was demonstrated? from my experience the answer is almost exclusively no; the problems tend to be complex, and their require a more robust solution.
So, I guess the bottom line would be - if you're looking for a one off solution, and open source is something you guys work with - JitterBit is definitely worth looking at, seeing if it's capable of helping out and has, indeed, a short learning curve (it would be important to look at maintenance, monitoring, trouble shooting, instance management etc)
If, however, you believe, as is often the case, that your solution would grow to become a BPM/integration platform in your organisation, and you need something more robust - I would put my money on BizTalk being a better candidate.
I've done a fair bit of integration with SAP, starting with the old SAP DCOM connector. More recently I've been involved in the selection of an integration platform to serve in an Enterprise Service Bus pattern.
We did web service samples to connect to SAP on a number of platforms, including BizTalk, Mule, Netweaver, Webmethods and Tibco. Webmethods won out based on licensing and capability, though BizTalk and Netweaver both had very high marks.
Jitterbit was not part of the evaluation - in fact I had to look it up to be sure I understood your question.
If your goal is just to be able to call an RFC, the .NET SAP connector works well.
If your goal is to expose a web service to wrapper a process in SAP, then BizTalk is good, but I recommend you see if your organization already has netweaver licensed as there are many web services available directly from SAP with no coding.
My recommendation is to avoid Jitterbug and Mule for the enterprise for now - unless Open Source is actually a popular thing at your place of employment. Netweaver and BizTalk are very robust, polished products.
If you are looking for something you can ship easily, then Jitterbug may make more sense. Though generally I'd recommend you define it as a web service call, and look to your customers technology stack for the most appropriate integration technique.
More context of what you are looking to achieve will enable a more accurate answer.
Michael,
We use Jitterbit in our organization and we've been very successful with it in various projects. Our SAP projects use XI and Jitterbit has dramatically simplified the ability to integrate web service interfaces with the various protocols it supports.
In addition to an excellent price (and we now subscribe to Jitterbit for support) we realize great value out of the support service. If we have any questions during our implementations they seem to provide all the subject matter expertise included in the support cost, so we're quite self sufficient.
We still have many other integration solutions in our company including VB and Java programs; it's a mess, but we don't believe that any one platform will meet all of our different divisions' needs. We have been using open source, specifically Linux and Apache for many years now, although IBM and Microsoft are also prevalent here.
We went with Jitterbit as it supports protocols needed to integrate any modern system and with SOA / Web Services being our stated direction Jitterbit was a great fit for what we needed.
Given that Jitterbit is Open Source, I would encourage you to download it and try it out.
I will say it simply, I have been using biztalk and was one of the people that helped validate the 2006 training course. Biztalk by far one the best server applications for Business process that is available today. You do also have to factor in the price point is ridiculously low compared to what else is out there.