We have a REST web service that receives requests from external systems and makes updates to our DB accordingly. I'm looking to implement a caching/queuing solution for the requests that come in, as we've had some DB server challenges lately, and have lost some messages when the DB server went down.
Before I start putting together a simple persistent file-based queue, I'm wanting to see if there are any good alternatives to JMS as it's use is restricted in our environment.
Current platforms:
Jboss 4.3
Richfaces 3.3
Spring 3.0.5
RESTEasy
** UPDATES **
Per skaffman's question below, my requirements for clustering, transactions, etc.
Clustering: Our web and app servers are all clustered, so the queue(s) will need to be able to process items from all cluster nodes. However, our commits are essentially atomic, so ordering and synchronization issues are extremely minimal. Thread and cluster-safety is not really a factor. Separate/Independent queues on each cluster would be sufficient.
Transactions: Again, due to the atomic nature of our data, transactional needs are minmal/not required outside of each individual request.
Security: Moderate concern, but I would anticipate that to be handled by our regular security on the Web Service. I wouldn't anticipate anything reading or writing to the queue(s) other than the web-app itself. That would only be necessary in instances of high volume or when the DB is unavailable.
Thanks,
Mike
For one project we did use a queue (HornetQ) but was integrated in the war and deployable on a Tomcat because the customer did not want Weblogic or JBoss application servers, but if your restricting policy goes to your application architecture as well such solution would be forbidden.
For another project we did not use any JMS implementation and we make the asynchronous implementation by using a message database and the Service Activator of the spring-integration framework for consuming the events.
That way any message publisher just insert a row in a DB table and the Service Activator trigs the event and call any other service (Spring, Web-service, etc...).
Related
Zookeeper plays several roles in the open-source workflow framework dolphinscheduler, such as heartbeat detection among masters and workers, task queue，event listener and distributed lock.
dolphin-sche framework
Is it possible to replace it by using database (mysql)? The main reason is to simplify the project structure .
zookeeper in DS is mainly used as:
Task queue, for master sending tasks to worker
Lock, for the communication between host(masters and workers)
Event watcher. Master listens the event that worker added or removed
it costs to replace zk as mysql.
zk mainly assumes the responsibility of the registry and monitors the application status. zk is very mature in this area and is a recognized solution in the industry. If MySQL wants to do this, the technical implementation cost will be larger, and may not achieve the desired effect.
BTW, their team is currently working on the SPI development for the registry, and in later versions, perhaps you can use other components, such as etcd, to achieve similar functionality.
for now, MasterServer and the WorkerServer nodes in the system all use the Zookeeper for cluster management and fault tolerance. In addition, the system also performs event monitoring and distributed locking based on ZooKeeper. We have also implemented queues based on Redis, but we hope that DolphinScheduler relies on as few components as possible, so we finally removed the Redis implementation.
so now DolphinScheduler can't work fine without Zookeeper, maybe in the future.
DolphinScheduler System Architecture:
For more documents please refer: Official Document.
I am investigating options for monitoring our installation in Swisscom's cloud-foundry. My objectives are the following:
monitor performance indicators for deployed application (such as cpu, disk, memory)
monitor performance indicators for services (slow queries, number of queries, ideally also some metrics on hitting quotas)
So far, I understand the options are the following (including some BUTs):
I used a very nice TOP cf-plugin (github)
This works very well. It seems that it registers itself to get the required firehose nozzles and consume data.
That is very useful for tracing / ad-hoc monitoring, but not very good for a serious infrastructure monitoring.
Another way I found is to use firehose-syslog solution.
This can be deployed as an app to (as far as I understand) do the job in similar way, as the TOP cf plugin.
The problem is, that it requires registered client, so it can authenticate with the doppler endpoint. For some reason, the top-cf-plugin does that automatically / in another way.
Last option i am considering is to build the monitoring itself to the App (using a special buildpack)
That can be for example done with Datadog. But it seems to also require a dedicated uaa client to register the Nozzle.
I would like to check, if somebody is (was) on the similar road, has some findings.
Eventually I would like to raise the following questions towards the swisscom community support:
is it possible to register uaac client to be able to ingest events through the firehose nozzle from external service? (this requires admin credentials if I was reading correctly)
is there an alternative way to authenticate with the nozzle (for example using a special user and his authentication token?)
is there any alternative to monitor the CF deployments in Swisscom? Eventually, is there a paper, blogpost or other form of documentation, that would be helpful in this respect (also for other users of AppCloud)?
Since it requires admin permissions, we can not give out UAA clients for the firehose.
However, there are different ways to get metrics in context of a user.
CF API
You can obtain basic metrics of a specific app by polling the CF API:
https://apidocs.cloudfoundry.org/5.0.0/apps/get_detailed_stats_for_a_started_app.html
However, since you have to poll (and for each app), it's not the recommended way.
Metrics in syslog drain
CF allows devs to forward their logs to syslog drains; in more recent versions, CF also sends metrics to this syslog drain (see https://docs.cloudfoundry.org/devguide/deploy-apps/streaming-logs.html#container-metrics).
For example, you could use Swisscom's Elasticsearch service to store these metrics and then analyze it using Kibana.
Metrics using loggregator (firehose)
The firehose allows streaming logs to clients for two types of roles:
Streaming all logs to admins (which requires a UAA client with admin permissions) and streaming app logs and metrics to devs with permissions in the app's space. This is also what the cf logs command uses. cf top also works this way (it enumerates all apps and streams the logs of each app).
However, you will find out that most open source tools that leverage the firehose only work in admin mode, since they're written for the platform operator.
Of course you also have the possibility to monitor your app by instrumenting it (white box approach), for example by configuring Spring actuator in a Spring boot app or by including an agent of your favourite APM vendor (Dynatrace, AppDynamics, ...)
I guess this is the most common approach; we've seen a lot of teams having success by instrumenting their applications. Especially since advanced monitoring anyway requires you to create your own metrics as the firehose provided cpu/memory metrics are not that powerful in a microservice world.
However, option 2. would be worth a try as well, especially since the ELK's stack metric support is getting better and better.
I'm doing a proof of concept at my company with Spring Cloud Config, using a GIT repository. Since I have a pretty fixed infrastructure, I can't have a RabbitMQ broker for Spring Cloud Bus, therefore I'm thinking of embedding the Spring Cloud Config Server in our apps and calling RefreshEndpoint.refresh() every couple of minutes.
One of the main things I'd like to know is what are the concurrency implications of calling RefreshEndpoint.refresh()? If there is a request already going on, could it have any issues?
I'm looking for specific product/technology or any proposed solution for the following problem:
I need a JTA-compliant transaction manager, that can enlist XAResources via resource-adapters and perform two-phase commit
It should be transparently available in JBoss AS/WildFly
It should be clustered with high-availability for
Transaction manager itself
Application server (JBoss) with applications as clients for TM deployed at AS
As "clustered" I mean not TM clustering, but client clustering sharing the same transaction: e.g. transaction begins on one JBoss server, then continues on second and is committed/rolled back on third. So the underlying resource (Database, enterprise bus, messaging) see all the requests from several app-servers as ONE transaction
As "high-availability" I mean that any component involved in transaction work execution could have a standby/hot-active instance that could complete/rollback work in case of main instance out of order. This include:
Transaction manager itself (it should not rely on one instance running, all transaction info should be replicated on-line within cluster)
Transaction clients (application running on JBoss instance which is processing transactonal call should fail-over on other JBoss instance in case of server outage)
I can't get the JTS catch in terms of work with XA resources (not in terms of work with saved transactional objects) and have not yet achieved any success in setting up JTS in cluster/HA. May be there is an issue that transaction could be managed by only one instance of TM and if it fails the transaction is buried until server restarted.
I don't know whether what I'm looking for is an utopia or whether I an not on the right way at all :)
I am thinking about porting a Spring Quartz based application to EJB 3.1 to see if EJB has improved. I am having problems understanding how fail-over works with the Schedule Timer Service. In Quartz, there are database tables which clustered Quartz instances use. If one node in your cluster crashes, jobs will still get executed on other nodes.
I have been looking at how the Timer Service persists things and it appears to use the file system of the server the Timer was created on. Is this true? I do not see how this would be possible as it would render the Timer Service unusable since it would not support failover.
So i must be missing something. Can anyone help me out with this?
The EJB timer service is simply not as advanced as Quartz (with or without Spring).
EJB timers are persisted to an unknown location. It may happen to be the file-system, but it could also be the Windows registry if you happen to be running on Windows, or it could be an LDAP server or whatever.
There was an issue on the EJB spec JIRA for some time about this, and it was discussed on the spec mailing list, but then it was brutally dropped and closed because no one bothered to reply anyone (perhaps because a lot of people were on vacation at the time). It's one of the lamest reasons to close an issue if you'd ask me, but I guess the spec lead sometimes must resort to such measures.
Anyway, in JBoss AS persisting happens to an embedded relational datasource, that on its turn writes to the filesystem. Via propriatary configuration you can point this datasource to any remote DB. Fail-over would have to come from propriatary JBoss functionality as well. Although EJB forbids lots of things for the sake of potential clustering, there's no explicit clustering support in the spec and thus specifically EJB timers are not cluster aware.
Not sure if this was available at the time of the question but you can use the 'cluster-ha-singleton' for this, it allows you to create a singleton timer that is invoked from a single cluster node, in case of failover of the chosen node a new node is elected to run the singleton (and therefore the timers)
http://www.jboss.org/quickstarts/eap/cluster-ha-singleton/
It mentions EAP but I am running on AS 7.2.0 fine, the jars are already included in /modules/org/jboss/