Hazelcast fails when multiple members disconnect from the cluster.
My scenario is so basic and my configuration has 3 bakcup option(it does not work). I have 4 members in a cluster and i use AtomicLong API to save my key->value. When all members are alive, everything is perfect. However, some data loss occurs when i kill 2 members at the same time(without waiting for a while). My member counts are always 4. is there any way to avoid this kind of data loss?
Config config = new Config();
NetworkConfig network = config.getNetworkConfig();
network.setPort(DistributedCacheData.getInstance().getPort());
config.getCacheConfig("default").setBackupCount(3);
JoinConfig join = network.getJoin();
join.getMulticastConfig().setEnabled(false);
join.getTcpIpConfig().setEnabled(true);
config.setNetworkConfig(network);
config.setInstanceName("member-name-here");
Thanks.
IAtomicLong has hard-coded 1 sync backup, you cannot configure it to have more than 1 backup. What you are doing is configuring Cache with 3 backups.
Below is a example demonstrates multiple member disconnect for IMap
Config config = new Config();
config.getMapConfig("myMap").setBackupCount(3);
HazelcastInstance[] instances = {
Hazelcast.newHazelcastInstance(config),
Hazelcast.newHazelcastInstance(config),
Hazelcast.newHazelcastInstance(config),
Hazelcast.newHazelcastInstance(config)
};
IMap<Integer, Integer> myMap = instances[0].getMap("myMap");
for (int i = 0; i < 1000; i++) {
myMap.set(i, i);
}
System.out.println(myMap.size());
instances[1].getLifecycleService().terminate();
instances[2].getLifecycleService().terminate();
System.out.println(myMap.size());
Related
No endpoint found for the service '{serviceB}' partition '{guid}' that matches the specified TargetReplicaSelector : 'RandomSecondaryReplica'
This is an error that has not always showed up, but it does sometimes.
I'm calling a stateful service B from another stateful service A, with service remoting, asking for a random secondary replica, to access state written to the primary.
I can see in Explorer that the partition is there and shows OK, and it has a primary and two ActiveSecondaries.
The service B has following:
protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
return new[] { new ServiceReplicaListener(context =>
this.CreateServiceRemotingListener(context), listenOnSecondary: true) };
}
I get all the partitions by this:
return Enumerable.Range(0, PartitionConstants.Partitions).Select(x =>
ServiceProxy.Create<IServiceB>(
ServiceBUri,
new ServicePartitionKey(x),
TargetReplicaSelector.RandomSecondaryReplica));
And the overall settings must be OK since sometimes it does work. And I know the primary is responding because I have saved state there.
So, what could cause this error when I can actually see the partition there, with the secondary replicas?
Update1 : Restarting the calling service made connection work. But they started together, and well after both had been running and working, the problem persisted, until I restarted. Howcome?
Update2 : This happens when whole cluster is started. At startup, Service A primaries calls Service B primaries for some registration. A polls B to know that it has initiated its internal state before doing this.
Then when this is complete, Service A goes on to check if its internal state needs update, and if so, it will call Service B again to retrieve state. Since it will not do any writing to B state, it calls secondary replicas. And here is when endpoint is not found.
When I restart Service A, endpoints are found.
Could it be that primaries are working and OK, but the secondaries are not yet OK?
How can I ascertain this? Is there some service fabric class that I can access to know whether the secondary will be found if I call for it?
Using a service primer found here, solved this issue. Seems like not all partition replicas was ready when being called.
Basically, what it does is counting all replicas of all partitions via FabricClient, until expected count is found.
Here is code:
public async Task WaitForStatefulService(Uri serviceInstanceUri, CancellationToken token)
{
StatefulServiceDescription description =
await this.Client.ServiceManager.GetServiceDescriptionAsync(serviceInstanceUri) as StatefulServiceDescription;
int targetTotalReplicas = description.TargetReplicaSetSize;
if (description.PartitionSchemeDescription is UniformInt64RangePartitionSchemeDescription)
{
targetTotalReplicas *= ((UniformInt64RangePartitionSchemeDescription)description.PartitionSchemeDescription).PartitionCount;
}
ServicePartitionList partitions = await this.Client.QueryManager.GetPartitionListAsync(serviceInstanceUri);
int replicaTotal = 0;
while (replicaTotal < targetTotalReplicas && !token.IsCancellationRequested)
{
await Task.Delay(this.interval);
//ServiceEventSource.Current.ServiceMessage(this, "CountyService waiting for National Service to come up.");
replicaTotal = 0;
foreach (Partition partition in partitions)
{
ServiceReplicaList replicaList = await this.Client.QueryManager.GetReplicaListAsync(partition.PartitionInformation.Id);
replicaTotal += replicaList.Count(x => x.ReplicaStatus == System.Fabric.Query.ServiceReplicaStatus.Ready);
}
}
}
I am using jmx to monitoring kafka topic.
val url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://broker1:9393/jmxrmi");
val jmxc = JMXConnectorFactory.connect(url, null);
val mbsc = jmxc.getMBeanServerConnection();
val messageCountObj = new ObjectName("kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=mytopic");
val messagesInPerSec = mbsc.getAttribute(messageCountObj,"MeanRate")
using this code I can get the MeanRate of "mytopic" on broker1.
but I have 10 brokers,how can I get the "mytopic"'s MeanRate from all my brokers?
I have try "service:jmx:rmi:///jndi/rmi://broker1:9393,broker2:9393,broker3:9393/jmxrmi"
got an error :(
It would be nice if it were that simple ;)
There's no way to do this as you outlined. You will need to make a seperate connection to each broker.
One possible solution would be to use MBeanServer Federation which would register proxies for each of your brokers in one MBeanServer, so if you did this on broker1, you could connect to service:jmx:rmi:///jndi/rmi://broker1:9393/jmxrmi and query the stats for all your brokers in one go, but you would need to query 10 different ObjectNames, query the value for each and then compute the MeanRate yourself. [Java] Pseudo code:
ObjectName wildcard = new ObjectName("*:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=mytopic");
double totalRate = 0d;
int respondingBrokers = 0;
for(ObjectName on : mbsc.queryNames(wildcard, null)) {
totalRate += (Double)mbsc.getAttribute(messageCountObj,"MeanRate");
respondingBrokers++;
}
// Average rate of mean rates: totalRate/respondingBrokers
Note: no exception handling, and I am assuming the rate type is a Double.
You could also create and register a custom MBean that computed the aggregate mean on the federated broker.
If you are maven oriented, you can build the OpenDMK from here.
I have a multiple partitioned stateful service. How can I enumerate all its partitions and aggregate results, using service remoting for communication between client and service?
You can enumerable the partitions using FabricClient:
var serviceName = new Uri("fabric:/MyApp/MyService");
using (var client = new FabricClient())
{
var partitions = await client.QueryManager.GetPartitionListAsync(serviceName);
foreach (var partition in partitions)
{
Debug.Assert(partition.PartitionInformation.Kind == ServicePartitionKind.Int64Range);
var partitionInformation = (Int64RangePartitionInformation)partition.PartitionInformation;
var proxy = ServiceProxy.Create<IMyService>(serviceName, new ServicePartitionKey(partitionInformation.LowKey));
// TODO: call service
}
}
Note that you should probably cache the results of GetPartitionListAsync since service partitions cannot be changed without recreating the service (you can just keep a list of the LowKey values).
In addition, FabricClient should also be shared as much as possible (see the documentation).
I have several remote nodes that stay on different computers and connected in cluster.
So, it is logging system on one of the nodes with role 'logging', that write logs in db.
I chose to use routing for delivering messages to logger from other nodes.
I have one node with main actor and three child actors. Each of them must send logs to logger node.
My configuration for router:
akka.actor.deployment {
/main/loggingRouter = {
router = adaptive-group
nr-of-instances = 100
cluster {
enabled = on
routees-path = "/user/loggingEvent"
use-role = logging
allow-local-routees = on
}
}
"/main/*/loggingRouter" = {
router = adaptive-group
nr-of-instances = 100
cluster {
enabled = on
routees-path = "/user/loggingEvent"
use-role = logging
allow-local-routees = on
}
}
}
And I create router in each actor with this code
val logging = context.actorOf(FromConfig.props(), name = "loggingRouter")
And send
logging ! LogProtocol("msg")
After that logger receives messages only from one child actor. I don't know how to debug it, but my guess that I apply wrong pattern for this.
What is the best practice for this task? Thx.
Actor from logger node:
system.actorOf(Logging.props(), name = "loggingEvent")
Problems is in routers with the same names. Ass I see, good pattern is to create one router in main actor and send it to children.
One way of creating JmDNS services is :
ServiceInfo.create(type, name, port, weight, priority, props);
where props is a Map which describes some propeties of the service. Does anybody have an example illustrating the use of theese properties, for instance how to use them in the reciever part.
I've tried :
Hashtable<String,String> settings = new Hashtable<String,String>();
settings.put("host", "hhgh");
settings.put("web_port", "hdhr");
settings.put("secure_web_port", "dfhdyhdh");
ServiceInfo info = ServiceInfo.create("_workstation._tcp.local.", "service6", 80, 0, 0, true, settings);
but, then in a machine receiving this service, what can I do to see those properties?
I would apreciate any help...
ServiceInfo info = jmDNS.getServiceInfo(serviceEvent.getType(), serviceEvent.getName());
Enumeration<String> ps = info.getPropertyNames();
while (ps.hasMoreElements()) {
String key = ps.nextElement();
String value = info.getPropertyString(key);
System.out.println(key + " " + value);
}
It has been a while since this was asked but I had the same question. One problem with the original question is that the host and ports should not be put into the text field, and in this case there should actually be two service types one secure and one insecure (or perhaps make use of subtypes).
Here is an incomplete example that gets a list of running workstation services:
ServiceInfo[] serviceInfoList = jmdns.list("_workstation._tcp.local.");
if(serviceInfoList != null) {
for (int index = 0; index < serviceInfoList.length; index++) {
int port = serviceInfoList[index].getPort();
int priority = serviceInfoList[index].getPriority();
int weight = serviceInfoList[index].getWeight();
InetAddress address = serviceInfoList[index].getInetAddresses()[0];
String someProperty = serviceInfoList[index].getPropertyString("someproperty");
// Build a UI or use some logic to decide if this service provider is the
// one you want to use based on prority, properties, etc.
...
}
}
Due to the way that JmDNS is implemented the first call to list() on a given type is slow (several seconds) but subsequent calls will be pretty fast. Providers of services can change the properties by calling info.setText(settings) and the changes will be propagated out to the listeners automatically.