i need to provide a H/A mechanism.
i understand thet zookeeper can be chosen as a leader election
i'm looking for the right pattern for this flow:
i need to implement a service that invoke a flow.
when it starting the flow [the flow is a looping flow], it must validate that it is the leader. (say by it's ip address).
i understand that i can put a value into a zookeeper that define that entering
instance, and dispose it when 1 loop end, or for a period of time.
it is that the right pattern?
also it seems that race condition issues if i use something like:
...
...
List<String> names = zk.getChildren( path, false );
String id = null;
// See whether we have already run for election in this process
for ( String name : names ) {
{
if ( name.startsWith( myIP ) ) {
id = name;
break;
}
}
if ( id == null ) {
id = zk.create( path + "/" + myIP, new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
}
boolean isLeader = id != null;
for example:
2 services read null
and than 2nd overrides the 1st signature, and both of them running the task.
can you help?
thanks
Using ZooKeeper correctly can be difficult and takes some studying of its APIs and semantics. There are some nice high-level libraries such as Curator that already have implemented common algorithms such as leader election. The ZooKeeper documentation also has a recipe for leader election.
Related
I'm trying to set the dead letter address for a queue via the JMS management API. From reading the latest Artemis docs it appears that I should be able to do this using the QueueControl.setDeadLetterAddress(...) method. See https://activemq.apache.org/artemis/docs/latest/management.html and search for "setDeadLetterAddress".
It is my understanding that the parameters of these methods should be found in the Artemis QueueControl javadocs here:
https://activemq.apache.org/artemis/docs/javadocs/javadoc-latest/org/apache/activemq/artemis/api/core/management/QueueControl.html
However, that documentation does not have any mention of a setDeadLetterAddress method or what parameters is might accept.
Does the QueueControl.setDeadLetterAddress method still exist and can it be called from the JMSManagementHelper.putOperationInvocation(...) method?
Many thanks!
Looking at the QueueControlImpl class code it is clear that the setDeadLetterAddress operation is no longer present. An operation in the ActiveMQServerControlImpl class named addAddressSettings does provide the capability to set the DLA for a queue (as well as plenty of other settings).
For example:
Queue managementQueue = ActiveMQJMSClient.createQueue("activemq.management");
Queue replyQueue = ActiveMQJMSClient.createQueue("management.reply");
JMSContext context = connectionFactory.createContext();
JMSConsumer consumer = context.createConsumer(replyQueue)) {
JMSProducer producer = context.createProducer();
producer.setJMSReplyTo(replyQueue);
// Using AddressSettings isn't required, but is provided
// for clarity.
AddressSettings settings = new AddressSettings()
.setDeadLetterAddress(new SimpleString("my.messages.dla"))
.setMaxDeliveryAttempts(5)
.setExpiryAddress(new SimpleString("ExpiryAddress"))
.setExpiryDelay(-1L) // No expiry
.setLastValueQueue(false)
.setMaxSizeBytes(-1) // No max
.setPageSizeBytes(10485760)
.setPageCacheMaxSize(5)
.setRedeliveryDelay(500)
.setRedeliveryMultiplier(1.5)
.setMaxRedeliveryDelay(2000)
.setRedistributionDelay(1000)
.setSendToDLAOnNoRoute(true)
.setAddressFullMessagePolicy(AddressFullMessagePolicy.PAGE)
.setSlowConsumerThreshold(-1) // No slow consumer checking
.setSlowConsumerCheckPeriod(1000)
.setSlowConsumerPolicy(SlowConsumerPolicy.NOTIFY)
.setAutoCreateJmsQueues(true)
.setAutoDeleteJmsQueues(false)
.setAutoCreateJmsTopics(true)
.setAutoDeleteJmsTopics(false)
.setAutoCreateQueues(true)
.setAutoDeleteQueues(false)
.setAutoCreateAddresses(true)
.setAutoDeleteAddresses(false);
Message m = context.createMessage();
JMSManagementHelper.putOperationInvocation(m, ResourceNames.BROKER, "addAddressSettings",
"my.messages",
settings.getDeadLetterAddress().toString(),
settings.getExpiryAddress().toString(),
settings.getExpiryDelay(),
settings.isLastValueQueue(),
settings.getMaxDeliveryAttempts(),
settings.getMaxSizeBytes(),
settings.getPageSizeBytes(),
settings.getPageCacheMaxSize(),
settings.getRedeliveryDelay(),
settings.getRedeliveryMultiplier(),
settings.getMaxRedeliveryDelay(),
settings.getRedistributionDelay(),
settings.isSendToDLAOnNoRoute(),
settings.getAddressFullMessagePolicy().toString(),
settings.getSlowConsumerThreshold(),
settings.getSlowConsumerCheckPeriod(),
settings.getSlowConsumerPolicy().toString(),
settings.isAutoCreateJmsQueues(),
settings.isAutoDeleteJmsQueues(),
settings.isAutoCreateJmsTopics(),
settings.isAutoDeleteJmsTopics(),
settings.isAutoCreateQueues(),
settings.isAutoDeleteQueues(),
settings.isAutoCreateAddresses(),
settings.isAutoDeleteAddresses());
producer.send(managementQueue, m);
Message response = consumer.receive();
// addAddressSettings returns void but this will also return errors if the
// method or parameters are wrong.
log.info("addAddressSettings Reply: {}", JMSManagementHelper.getResult(response));
No endpoint found for the service '{serviceB}' partition '{guid}' that matches the specified TargetReplicaSelector : 'RandomSecondaryReplica'
This is an error that has not always showed up, but it does sometimes.
I'm calling a stateful service B from another stateful service A, with service remoting, asking for a random secondary replica, to access state written to the primary.
I can see in Explorer that the partition is there and shows OK, and it has a primary and two ActiveSecondaries.
The service B has following:
protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
return new[] { new ServiceReplicaListener(context =>
this.CreateServiceRemotingListener(context), listenOnSecondary: true) };
}
I get all the partitions by this:
return Enumerable.Range(0, PartitionConstants.Partitions).Select(x =>
ServiceProxy.Create<IServiceB>(
ServiceBUri,
new ServicePartitionKey(x),
TargetReplicaSelector.RandomSecondaryReplica));
And the overall settings must be OK since sometimes it does work. And I know the primary is responding because I have saved state there.
So, what could cause this error when I can actually see the partition there, with the secondary replicas?
Update1 : Restarting the calling service made connection work. But they started together, and well after both had been running and working, the problem persisted, until I restarted. Howcome?
Update2 : This happens when whole cluster is started. At startup, Service A primaries calls Service B primaries for some registration. A polls B to know that it has initiated its internal state before doing this.
Then when this is complete, Service A goes on to check if its internal state needs update, and if so, it will call Service B again to retrieve state. Since it will not do any writing to B state, it calls secondary replicas. And here is when endpoint is not found.
When I restart Service A, endpoints are found.
Could it be that primaries are working and OK, but the secondaries are not yet OK?
How can I ascertain this? Is there some service fabric class that I can access to know whether the secondary will be found if I call for it?
Using a service primer found here, solved this issue. Seems like not all partition replicas was ready when being called.
Basically, what it does is counting all replicas of all partitions via FabricClient, until expected count is found.
Here is code:
public async Task WaitForStatefulService(Uri serviceInstanceUri, CancellationToken token)
{
StatefulServiceDescription description =
await this.Client.ServiceManager.GetServiceDescriptionAsync(serviceInstanceUri) as StatefulServiceDescription;
int targetTotalReplicas = description.TargetReplicaSetSize;
if (description.PartitionSchemeDescription is UniformInt64RangePartitionSchemeDescription)
{
targetTotalReplicas *= ((UniformInt64RangePartitionSchemeDescription)description.PartitionSchemeDescription).PartitionCount;
}
ServicePartitionList partitions = await this.Client.QueryManager.GetPartitionListAsync(serviceInstanceUri);
int replicaTotal = 0;
while (replicaTotal < targetTotalReplicas && !token.IsCancellationRequested)
{
await Task.Delay(this.interval);
//ServiceEventSource.Current.ServiceMessage(this, "CountyService waiting for National Service to come up.");
replicaTotal = 0;
foreach (Partition partition in partitions)
{
ServiceReplicaList replicaList = await this.Client.QueryManager.GetReplicaListAsync(partition.PartitionInformation.Id);
replicaTotal += replicaList.Count(x => x.ReplicaStatus == System.Fabric.Query.ServiceReplicaStatus.Ready);
}
}
}
I am using jmx to monitoring kafka topic.
val url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://broker1:9393/jmxrmi");
val jmxc = JMXConnectorFactory.connect(url, null);
val mbsc = jmxc.getMBeanServerConnection();
val messageCountObj = new ObjectName("kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=mytopic");
val messagesInPerSec = mbsc.getAttribute(messageCountObj,"MeanRate")
using this code I can get the MeanRate of "mytopic" on broker1.
but I have 10 brokers,how can I get the "mytopic"'s MeanRate from all my brokers?
I have try "service:jmx:rmi:///jndi/rmi://broker1:9393,broker2:9393,broker3:9393/jmxrmi"
got an error :(
It would be nice if it were that simple ;)
There's no way to do this as you outlined. You will need to make a seperate connection to each broker.
One possible solution would be to use MBeanServer Federation which would register proxies for each of your brokers in one MBeanServer, so if you did this on broker1, you could connect to service:jmx:rmi:///jndi/rmi://broker1:9393/jmxrmi and query the stats for all your brokers in one go, but you would need to query 10 different ObjectNames, query the value for each and then compute the MeanRate yourself. [Java] Pseudo code:
ObjectName wildcard = new ObjectName("*:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=mytopic");
double totalRate = 0d;
int respondingBrokers = 0;
for(ObjectName on : mbsc.queryNames(wildcard, null)) {
totalRate += (Double)mbsc.getAttribute(messageCountObj,"MeanRate");
respondingBrokers++;
}
// Average rate of mean rates: totalRate/respondingBrokers
Note: no exception handling, and I am assuming the rate type is a Double.
You could also create and register a custom MBean that computed the aggregate mean on the federated broker.
If you are maven oriented, you can build the OpenDMK from here.
Our application is build upon mongodb replica set.
I'd like to catch all exceptions thrown among the time frame when replica set is in process of automatic failover.
I will make application retry or wait for failover completes.
So that the failover won't influence user.
I found document describing the behavior of java driver here: https://jira.mongodb.org/browse/DOCS-581
I write a test program to find all possible exceptions, they are all MongoException but with different message:
MongoException.Network: "Read operation to server /10.11.0.121:27017 failed on database test"
MongoException: "can't find a master"
MongoException: "not talking to master and retries used up"
MongoException: "No replica set members available in [ here is replica set status ] for { "mode" : "primary"}"
Maybe more...
I'm confused and not sure if it is safe to determine by error message.
Also I don't want to catch all MongoException.
Any suggestion?
Thanks
I am now of the opinion that Mongo in Java is particularly weak in this regards. I don't think your strategy of interpreting the error codes scales well or will survive driver evolution. This is, of course, opinion.
The good news is that the Mongo driver provides a way get the status of a ReplicaSet: http://api.mongodb.org/java/2.11.1/com/mongodb/ReplicaSetStatus.html. You can use it directly to figure out whether there is a Master visible to your application. If that is all you want to know, the http://api.mongodb.org/java/2.11.1/com/mongodb/Mongo.html#getReplicaSetStatus() is all you need. Grab that kid and check for a not-null master and you are on your way.
ReplicaSetStatus rss = mongo.getReplicaSetStatus();
boolean driverInFailover = rss.getMaster() == null;
If what you really need is to figure out if the ReplSet is dead, read-only, or read-write, this gets more difficult. Here is the code that kind-of works for me. I hate it.
#Override
public ReplSetStatus getReplSetStatus() {
ReplSetStatus rss = ReplSetStatus.DOWN;
MongoClient freshClient = null;
try {
if ( mongo != null ) {
ReplicaSetStatus replicaSetStatus = mongo.getReplicaSetStatus();
if ( replicaSetStatus != null ) {
if ( replicaSetStatus.getMaster() != null ) {
rss = ReplSetStatus.ReadWrite;
} else {
/*
* When mongo.getReplicaSetStatus().getMaster() returns null, it takes a a
* fresh client to assert whether the ReplSet is read-only or completely
* down. I freaking hate this, but take it up with 10gen.
*/
freshClient = new MongoClient( mongo.getAllAddress(), mongo.getMongoClientOptions() );
replicaSetStatus = freshClient.getReplicaSetStatus();
if ( replicaSetStatus != null ) {
rss = replicaSetStatus.getMaster() != null ? ReplSetStatus.ReadWrite : ReplSetStatus.ReadOnly;
} else {
log.warn( "freshClient.getReplicaSetStatus() is null" );
}
}
} else {
log.warn( "mongo.getReplicaSetStatus() returned null" );
}
} else {
throw new IllegalStateException( "mongo is null?!?" );
}
} catch ( Throwable t ) {
log.error( "Ingore unexpected error", t );
} finally {
if ( freshClient != null ) {
freshClient.close();
}
}
log.debug( "getReplSetStatus(): {}", rss );
return rss;
}
I hate it because it doesn't follow the Mongo Java Driver convention of your application only needs a single Mongo and through this singleton you connect to the rest of the Mongo data structures (DB, Collection, etc). I have only been able to observe this working by new'ing up a second Mongo during the check so that I can rely upon the ReplicaSetStatus null check to discriminate between "ReplSet-DOWN" and "read-only".
What is really needed in this driver is some way to ask direct questions of the Mongo to see if the ReplSet can be expected at this moment to support each of the WriteConcerns or ReadPreferences. Something like...
/**
* #return true if current state of Client can support readPreference, false otherwise
*/
boolean mongo.canDoRead( ReadPreference readPreference )
/**
* #return true if current state of Client can support writeConcern; false otherwise
*/
boolean mongo.canDoWrite( WriteConcern writeConcern )
This makes sense to me because it acknowledges the fact that the ReplSet may have been great when the Mongo was created, but conditions right now mean that Read or Write operations of a specific type may fail due to changing conditions.
In any event, maybe http://api.mongodb.org/java/2.11.1/com/mongodb/ReplicaSetStatus.html gets you what you need.
When Mongo is failing over, there are no nodes in a PRIMARY state. You can just get the replica set status via the replSetGetStatus command and look for a master node. If you don't find one, you can assume that the cluster is in a failover transition state, and can retry as desired, checking the replica set status on each failed connection.
I don't know the Java driver implementation itself, but I'd do catch all MongoExceptions, then filter them on getCode() basis. If the error code does not apply to replica sets failures, then I'd rethrow the MongoException.
The problem is, to my knowledge there is no error codes reference in the documentation. Well there is a stub here, but this is fairly incomplete. The only way is to read the code of the Java driver to know what code it uses…
One way of creating JmDNS services is :
ServiceInfo.create(type, name, port, weight, priority, props);
where props is a Map which describes some propeties of the service. Does anybody have an example illustrating the use of theese properties, for instance how to use them in the reciever part.
I've tried :
Hashtable<String,String> settings = new Hashtable<String,String>();
settings.put("host", "hhgh");
settings.put("web_port", "hdhr");
settings.put("secure_web_port", "dfhdyhdh");
ServiceInfo info = ServiceInfo.create("_workstation._tcp.local.", "service6", 80, 0, 0, true, settings);
but, then in a machine receiving this service, what can I do to see those properties?
I would apreciate any help...
ServiceInfo info = jmDNS.getServiceInfo(serviceEvent.getType(), serviceEvent.getName());
Enumeration<String> ps = info.getPropertyNames();
while (ps.hasMoreElements()) {
String key = ps.nextElement();
String value = info.getPropertyString(key);
System.out.println(key + " " + value);
}
It has been a while since this was asked but I had the same question. One problem with the original question is that the host and ports should not be put into the text field, and in this case there should actually be two service types one secure and one insecure (or perhaps make use of subtypes).
Here is an incomplete example that gets a list of running workstation services:
ServiceInfo[] serviceInfoList = jmdns.list("_workstation._tcp.local.");
if(serviceInfoList != null) {
for (int index = 0; index < serviceInfoList.length; index++) {
int port = serviceInfoList[index].getPort();
int priority = serviceInfoList[index].getPriority();
int weight = serviceInfoList[index].getWeight();
InetAddress address = serviceInfoList[index].getInetAddresses()[0];
String someProperty = serviceInfoList[index].getPropertyString("someproperty");
// Build a UI or use some logic to decide if this service provider is the
// one you want to use based on prority, properties, etc.
...
}
}
Due to the way that JmDNS is implemented the first call to list() on a given type is slow (several seconds) but subsequent calls will be pretty fast. Providers of services can change the properties by calling info.setText(settings) and the changes will be propagated out to the listeners automatically.