Configure Spring Data Redis to perform all operations via Elasticache configuration endpoint? - spring-data

Description
Is it possible for Spring Data Redis to use Elasticache's configuration endpoint to perform all cluster operations (i.e., reading, writing, etc.)?
Long Description
I have a Spring Boot application that uses a Redis cluster as data store. The Redis cluster is hosted on AWS Elasticache running in cluster-mode enabled. The Elasticache cluster has 3 shards spread out over 12 nodes. The Redis version that the cluster is running is 6.0.
The service isn't correctly writing or retrieving data from the cluster. Whenever performing any of these operations, I get a message similar to the following:
io.lettuce.core.RedisCommandExecutionException: MOVED 16211 10.0.7.254:6379
In searching the internet, it appears that the service isn't correctly configured for a cluster. The fix seems to be set the spring.redis.cluster.nodes property with a list of all the nodes in the Elasticache cluster (see here and here). I find this rather needless, considering that the Elasticache configuration endpoint is supposed to be used for all read and write operations (see the "Finding Endpoints for a Redis (Cluster Mode Enabled) Cluster" section here).
My question is this: can Spring Data Redis use Elasticache's configuration endpoint to perform all reads and writes, the way the AWS documentation describes? I'd rather not hand over a list of all the nodes if Spring Data Redis can use the configuration endpoint the way its meant to be used. This seems like a serious limitation to me.
Thanks in advance!

Here is what I found works:
#Bean
public RedisConnectionFactory lettuceConnectionFactory()
{
LettuceClientConfiguration config =
LettucePoolingClientConfiguration
.builder()
.*your configuration settings*
.build();
RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration();
clusterConfig.addClusterNode(new RedisNode("xxx.v1tc03.clustercfg.use1.cache.amazonaws.com", 6379));
return new LettuceConnectionFactory(clusterConfig, config);
}
where xxx is the name from your elasticache cluster.

Related

Implementing multi-datacenter Cassandra with Phantom driver

I'm working with Cassandra 3.x and Phantom driver (scala),
and modifying my Cassandra deployment from a simple, three nodes cluster to a multi datacenter Cassandra deployment that consists of two datacenters:
Transactional - the "main" datacenter, to which all reads/writes occur (except for reads/writes done by some analytics job).
Analytics - a datacenter used for analytics purposes only. The analytics job should operate (i.e. read/write to) on this datacenter.
Both datacenters are configured with the proper snitch and replication factor strategies.
Based on this article ("Workload Separation" section), I'm supposed to be able to read/write from the "Transactional" datacenter, and run analytics jobs on the "Analytics" datacenter however, I'm not sure how to get this to work with the phantom driver.
How can I configure the driver to read/write from the proper datacenter?
Will setting the hosts in ContactPoints class to nodes from the Transactional datacenter only do the trick?
By default, Java driver 3.x uses so-called DCAware load balancing policy combined with TokenAware policy. Data center could be configured explicitly by using withLocalDc function of builder, but it could be omitted and driver will use the datacenter of the first contact point that was reached at initialization. So you can just point Phantom only to servers in the transactional DC, and it will work only with it (until you're using non-local consistency levels, such as QUORUM/SERIAL, EACH_QUORUM, etc.)

Quartz cluster and redis jobStore

Is there any redis jobStore able to support a quartz cluster?
Have anybody been able to build that?
By other side, what's exactly a quartz cluster? I mean, is it able to have two services running the same quartz.properties file pointing to a redis?
EDIT
I've tried with this redis job store but it seems doesn't supprt quartz clustering:
JobStore class 'net.joelinn.quartz.jobstore.RedisJobStore' props could not be configured. [See nested exception: java.lang.NoSuchMethodException: No setter for property 'isClustered']
quartz.properties:
org.quartz.scheduler.instanceName=office-scheduler-service
org.quartz.scheduler.instanceId=AUTO
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=20000
# thread-pool
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=2
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread=true
org.quartz.jobStore.class = net.joelinn.quartz.jobstore.RedisJobStore
org.quartz.jobStore.host = redisbo
org.quartz.jobStore.misfireThreshold = 60000
you don't need to configure cluster, please check the source code, it is already clustered
Quartz JDBC documentation explains how it handles executing jobs in a cluster of application nodes. RedisJobStore extended that to utilize the Redis storage, and it will work in a cluster mode (Quartz cluster - not Redis cluster) by default without requiring you to enable that.
Basically Quartz uses a shared database to record which scheduler instance is currently working on a job, as opposed to direct node communication among application schedulers. When a scheduler instance picks up a job, it safely registers its instance id with the running job and persists it in the database. This support by the job store is evident in the schema used by RedisJobStore, indicated by the blocked_by fields.

Does AWS Aurura Postgresql do automatic query splitting

I am trying to work out if Aurora Postgres cluster endpoint does automatic query read write splitting between the reader and writer/s? Or to I need to use something like pgpool2 for the splitting?
I have attempted pushing read/writes at it, but it looks like only the reader is being hit?
but it looks like only the reader is being hit?
Only the writer should be hit via the cluster endpoint.
The cluster endpoint connects you to the primary instance for the DB cluster. You can perform both read and write operations using the cluster endpoint.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Connecting.html
Aurora does not do read/write splitting. It provides you with a cluster endpoint, which automatically points to the cluster's current writer, and a read-only endpoint, which connects you to any one of the readers.
If the cluster only has one instance (Aurora doesn't technically require you to have any readers, but recovery in the event of a failure would require much more time, since the one instance would have to be rebuilt by the system if it failed -- but no data would be lost) then both endpoints take you to the writer.

How to make the Eureka server strong?

I am new to Spring Cloud. Currently, I want to build a new micro service based on Spring Cloud. It is very easy to build a new Eureka server. But my question is that how to make it high availability ? For example I create two Eureka server and a load balancer. When one of the Eureka server is down, the system still works well. But I don't know to to consist registered information in the two Eureka server.
I have already asked something similar in the spring cloud gitter channel.
Because of the CAP theorem, something as a distributes Service discovery has to decide, either to provide availability, or more consistency, with a trade off to the other one.
in short, by quoting Spencer Gibb:
Eureka favors availability over consistency
so it is very available, while registred services may be not acutal anymore.
As Spencer suggested, if consistency is something you need more then availability, try Consul together with spring cloud consul intead

spring cloud consul service names

I am switching all my service infrastructure from eureka to consul.
In the eureka case I have multiple services with the same name and Eureka handles this via the Application and instance to differentiate.
In the consul case, if I have this naming scheme, does spring cloud generate unique ids under eh covers?
I read where consul will use the id and name synonymously unless you register them under unique ids.
So you can have service 1 as (name=myservice, id=xxx) and service 2 as (name=myservice, id=yyy).
So in that way consul preserves uniqueness. What does spring cloud do under the covers?
Ok, so it appears that the question is not clear.
I know that I can specify uniqueness when I define them but I don't
I have a large microservices-based system in production. We have multiples of each microservices for both redundancy and scaling and we do not specifically set uniqueness on the services.
We don't because Eureka does this for us. Say I have a CustomerAccountService with 5 instances then I when I request customer account service I can see 5 instances. Looking at the Eureka data model, we see one Application and 5 instances of it.
So I am planning on moving to consul and want t preserve a similar mode of operation. Many instances of the same time of service.
What I really want to know is how the spring consul registration works under the covers or do I have to do something special for this.
I do know that COnsul defines a name and an id and that they can be the same or they can be different.
So can I have the name for 5 instances the same and have the id variate? If so, how does that happen in the spring cloud consul version of this.
Any application registered with the same spring.application.name in Consul using Spring Cloud will be grouped together just like Eureka.