Data is not getting replicated to each clickhouse replica. While checking clickhouse-server log it is showing
DB::StorageReplicatedMergeTree::queueTask()::: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Host not found: ip-172-1-140-243 (version 19.9.2.4)
I have 3 different machines on each machine I've installed both clickhouse and zookeeper. I'm trying to setup 3 node clickhouse cluster using zookeeper. I have followed these steps for configuration https://blog.uiza.io/replicated-and-distributed-on-clickhouse-part-2/ . I have created tables and replicas on all clickhouse instances and verified in zookeeper as well. Directory got created in zookeeper for all replicas. Have created /etc/metrica.xml , zoo.cfg , /etc/clickhouse-server/config.xml on all instances. Providing all 3 files from 1 instance
/etc/metrica.xml file
<?xml version="1.0"?>
<yandex>
<clickhouse_remote_servers>
<perftest_3shards_1replicas>
<shard>
<internal_replication>true</internal_replication>
<replica>
<default_database>dwh01</default_database>
<host>172.1.34.199</host>
<port>9000</port>
</replica>
<replica>
<default_database>dwh01</default_database>
<host>172.1.73.156</host>
<port>9000</port>
</replica>
</shard>
<shard>
<internal_replication>true</internal_replication>
<replica>
<default_database>dwh02</default_database>
<host>172.1.73.156</host>
<port>9000</port>
</replica>
<replica>
<default_database>dwh02</default_database>
<host>172.1.140.243</host>
<port>9000</port>
</replica>
</shard>
<shard>
<internal_replication>true</internal_replication>
<replica>
<default_database>dwh03</default_database>
<host>172.1.140.243</host>
<port>9000</port>
</replica>
<replica>
<default_database>dwh03</default_database>
<host>172.1.34.199</host>
<port>9000</port>
</replica>
</shard>
</perftest_3shards_1replicas>
</clickhouse_remote_servers>
<zookeeper-servers>
<node index="1">
<host>172.1.34.199</host>
<port>2181</port>
</node>
<node index="2">
<host>172.1.73.156</host>
<port>2181</port>
</node>
<node index="3">
<host>172.1.140.243</host>
<port>2181</port>
</node>
</zookeeper-servers>
<macros replace="replace">
<cluster>OLAPLab</cluster>
<dwhshard00>01</dwhshard00>
<dwhshard01>03</dwhshard01>
<dwhreplica00>01</dwhreplica00>
<dwhreplica01>02</dwhreplica01>
<shard>01</shard>
<replica>node1</replica>
</macros>
<interserver_http_host>ip-172-1-34-199</interserver_http_host>
</yandex>
/etc/clickhouse-server/config.xml
Only added this line rest of the config is default config
<listen_host>::</listen_host>
/usr/lib/zookeeper/conf/zoo.cfg
maxClientCnxns=50
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/var/lib/zookeeper
# the port at which the clients will connect
clientPort=2181
# the directory where the transaction logs are stored.
dataLogDir=/var/lib/zookeeper
server.1=172.1.34.199:2888:3888
server.2=172.1.73.156:2888:3888
server.3=172.1.140.243:2888:3888
/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost6 localhost6.localdomain6
127.0.0.1 ip-172-1-34-199
127.0.0.1 172.1.34.199
What should I change to get replicated data in all replicas belongs to all instances
It looks like you need to add to hosts-file of both nodes (172.1.34.199 and 172.1.73.156) the string
172.1.140.243 ip-172-1-140-243
Are the clickhouse and zookeeper ports accessible among the machines ?
As in, can you run wget http://172.1.140.243:9000 ? Same applies to the zookeper ports respectively.
Related
I am running Rundeck 3.0.20-20190408 in Red Hat Enterprise Linux 8 and I am connecting the node using pem key.
resource.xml
<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="10.10.101.100" description="MyServer" tags="" hostname="10.10.101.100" osArch="amd64" osFamily="unix" osName="Linux" osVersion="3.10.0-693.2.2.el7.x86_64" username="rundeckuser" ssh-keypath="/home/username#domain.com/rundeckuser.pem" />
</project>
Getting below error while executing the commands
Execution failed: 3138 in project test_default: [Workflow result: ,
step failures: {1=Dispatch failed on 1 nodes: [10.10.101.100:
ConfigurationFailure: SSH Keyfile does not exist:
/home/username#domain.com/rundeckuser.pem +
{dataContext=MultiDataContextImpl(map={}, base=null)} ]}, Node
failures: {10.10.101.100=[ConfigurationFailure: SSH Keyfile does not
exist: /home/username#domain.com/rundeckuser.pem +
{dataContext=MultiDataContextImpl(map={}, base=null)} ]}, status:
failed]
ssh-keypath must be defined with a local path reachable by rundeck user (or the user that launches Rundeck if you're using a WAR based installation), like this example.
But the best approach is to use the Rundeck's key storage (adding the rundeck server public key to your target servers to access it, follow this if you like). Also, you can see the best practices here (it's using Rundeck 2.11 but is the same principle for all versions.
I want to run two deployed applications ( .ear ) in two instances of JBoss 6.0 at the same time
I have changed all used ports of both standalone.xml files including http , management-http , etc...
Like this:
application1.ear : socket-binding name="http" port="8080
application2.ear : socket-binding name="http" port="8081
application1.ear : name="management-http" port="9990
application2.ear : name="management-http" port="9991
Any Help is appreciated
Following are the two ways to run mutliple JBoss instance on same server.
Bind each instance to a different IP address
This is the easiest and most recommended way to solve this problem. If the server has multiple NICs then this is simple. If not, then one must "multi-home" the server. In other words, assign the server more than one IP address through OS configuration. Start the instances like so:
$JBOSS_HOME1/bin/run.sh -b <ip-addr-1>
$JBOSS_HOME2/bin/run.sh -b <ip-addr-2>
The same $JBOSS_HOME can be used with multiple "profiles" in $JBOSS_HOME/server. For example:
$JBOSS_HOME/bin/run.sh -b <ip-addr-1> -c node1
$JBOSS_HOME/bin/run.sh -b <ip-addr-2> -c node2
Service Binding Manager
Configure the "Service Binding Manager" to tell the JBoss instances which ports to use.
Uncomment the "jboss.system:service=ServiceBindingManager" MBean in $JBOSS_HOME/server/$PROFILE/conf/jboss-service.xml.
<mbean code="org.jboss.services.binding.ServiceBindingManager"
name="jboss.system:service=ServiceBindingManager">
<attribute name="ServerName">ports-01</attribute>
<attribute name="StoreURL">${jboss.home.url}/docs/examples/binding-manager/sample-bindings.xml</attribute>
<attribute name="StoreFactoryClassName">
org.jboss.services.binding.XMLServicesStoreFactory
</attribute>
</mbean>
This tells JBoss to use the port numbering scheme defined by "ports-01" in $JBOSS_HOME/docs/examples/binding-manager/sample-bindings.xml. This scheme increases the second most significant digit of every port by 100. For example, the JNDI port is 1099 by default but 1199 using the ports-01 scheme; the HTTP port is 8080 by default but 8180 using the ports-01 scheme. The sample-bindings.xml file contains 4 port schemes:
ports-default
ports-01
ports-02
ports-03
You may want to configure the port set used at start up from the command line or through a system property. If so, adjust the MBean's ServerName to refer to a system property, for example:
<mbean code="org.jboss.services.binding.ServiceBindingManager"
name="jboss.system:service=ServiceBindingManager">
<attribute name="ServerName">${jboss.service.binding.set:ports-default}</attribute>
<attribute name="StoreURL">${jboss.home.url}/docs/examples/binding-manager/sample-bindings.xml</attribute>
<attribute name="StoreFactoryClassName">
org.jboss.services.binding.XMLServicesStoreFactory
</attribute>
</mbean>
Now change it through the following property directly on run.sh/run.bat or add it to your run.conf options:
-Djboss.service.binding.set=ports-01
If you need more than 4 port sets defined in sample-bindings.xml by default, please refer to the following article for JBOSS 6 EAP:
https://access.redhat.com/site/solutions/237933
We are currently testing to move from Wildfly 8.2.0 to Wildfly 9.0.0.CR1 (or CR2 built from snapshot). The system is a cluster using mod_cluster and is running on VPS what in fact prevents it from using multicast.
On 8.2.0 we have been using the following configuration of the modcluster that works well:
<mod-cluster-config proxy-list="1.2.3.4:10001,1.2.3.5:10001" advertise="false" connector="ajp">
<dynamic-load-provider>
<load-metric type="cpu"/>
</dynamic-load-provider>
</mod-cluster-config>
Unfortunately, on 9.0.0 proxy-list was deprecated and the start of the server will finish with an error. There is a terrible lack of documentation, however after a couple of tries I have discovered that proxy-list was replaced with proxies that are a list of outbound-socket-bindings. Hence, the configuration looks like the following:
<mod-cluster-config proxies="mc-prox1 mc-prox2" advertise="false" connector="ajp">
<dynamic-load-provider>
<load-metric type="cpu"/>
</dynamic-load-provider>
</mod-cluster-config>
And the following should be added into the appropriate socket-binding-group (full-ha in my case):
<outbound-socket-binding name="mc-prox1">
<remote-destination host="1.2.3.4" port="10001"/>
</outbound-socket-binding>
<outbound-socket-binding name="mc-prox2">
<remote-destination host="1.2.3.5" port="10001"/>
</outbound-socket-binding>
So far so good. After this, the httpd cluster starts registering the nodes. However I am getting errors from load balancer. When I look into /mod_cluster-manager, I see a couple of Node REMOVED lines and there are also many many errors like:
ERROR [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000042: Error MEM sending STATUS command to node1/1.2.3.4:10001, configuration will be reset: MEM: Can't read node
In the log of mod_cluster there are the equivalent warnings:
manager_handler STATUS error: MEM: Can't read node
As far as I understand, the problem is that although wildfly/modcluster is able to connect to httpd/mod_cluster, it does not work the other way. Unfortunately, even after an extensive effort I am stuck.
Could someone help with setting mod_cluster for Wildfly 9.0.0 without advertising? Thanks a lot.
I ran into the Node Removed issue to.
I managed to solve it by using the following as instance-id
<subsystem xmlns="urn:jboss:domain:undertow:2.0" instance-id="${jboss.server.name}">
I hope this will help someone else to ;)
There is no need for any unnecessary effort or uneasiness about static proxy configuration. Each WildFly distribution comes with xsd sheets that describe xml subsystem configuration. For instance, with WildFly 9x, it's:
WILDFLY_DIRECTORY/docs/schema/jboss-as-mod-cluster_2_0.xsd
It says:
<xs:attribute name="proxies" use="optional">
<xs:annotation>
<xs:documentation>List of proxies for mod_cluster to register with defined by outbound-socket-binding in socket-binding-group.</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:list itemType="xs:string"/>
</xs:simpleType>
</xs:attribute>
The following setup works out of box
Download wildfly-9.0.0.CR1.zip or build with ./build.sh from sources
Let's assume you have 2 boxes, Apache HTTP Server with mod_cluster acting as a load balancing proxy and your WildFly server acting as a worker. Make sure botch servers can access each other on both MCMP enabled VirtualHost's address and port (Apache HTTP Server side) and on WildFly AJP and HTTP connector side. The common mistake is to binf WildFLy to localhost; it then reports its addess as localhost to the Apache HTTP Server residing on a dofferent box, which makes it impossible for it to contact WildFly server back. The communication is bidirectional.
This is my configuration diff from the default wildfly-9.0.0.CR1.zip.
328c328
< <mod-cluster-config advertise-socket="modcluster" connector="ajp" advertise="false" proxies="my-proxy-one">
---
> <mod-cluster-config advertise-socket="modcluster" connector="ajp">
384c384
< <subsystem xmlns="urn:jboss:domain:undertow:2.0" instance-id="worker-1">
---
> <subsystem xmlns="urn:jboss:domain:undertow:2.0">
435c435
< <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:102}">
---
> <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
452,454d451
< <outbound-socket-binding name="my-proxy-one">
< <remote-destination host="10.10.2.4" port="6666"/>
< </outbound-socket-binding>
456c453
< </server>
---
> </server>
Changes explanation
proxies="my-proxy-one", outbound socket binding name; could be more of them here.
instance-id="worker-1", the name of the worker, a.k.a. JVMRoute.
offset -- you could ignore, it's just for my test setup. Offset does not apply to outbound socket bindings.
<outbound-socket-binding name="my-proxy-one"> - IP and port of the VirtualHost in Apache HTTP Server containing EnableMCPMReceive directive.
Conclusion
Generally, these MEM read / node error messages are related to network problems, e.g. WildFly can contact Apache, but Apache cannot contact WildFly back. Last but not least, it could happen that the Apache HTTP Server's configuration uses PersistSlots directive and some substantial enviroment conf change took place, e.g. switch from mpm_prefork to mpm_worker. In this case, MEM Read error messages are not realted to WildFly, but to the cached slotmem files in HTTPD/cache/mod_custer that need to be deleted.
I'm certain it's network in your case though.
After a couple of weeks I got back to the problem and found the solution. The problem was - of course - in configuration and had nothing in common with the particular version of Wildfly. Mode specifically:
There were three nodes in the domain and three servers in each node. All nodes were launched with the following property:
-Djboss.node.name=nodeX
...where nodeX is the name of a particular node. However, it meant that all three servers in the node get the same name, which is exactly what confused the load balancer.
As soon as I have removed this property, everything started to work.
There are two scenarios as mentioned below.
First Scenario:
Node1: 192.168.69.201
Node2: 192.168.69.202
Second Scenario:
Node1: 192.168.69.201
Node2: 192.168.68.203
Cluster is formed successfully in the first scenario but not in the second one. I used TCP protocol as mentioned in the below xml:
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-3.2.xsd">
<TCP
bind_addr="${jgroups.tcp.address:192.168.68.201}"
bind_port="${jgroups.tcp.port:7800}"
loopback="true"
...
/>
<MPING bind_addr="${jgroups.bind_addr:192.168.68.201}" break_on_coord_rsp="true"
mcast_addr="${jgroups.mping.mcast_addr:228.2.4.6}"
mcast_port="${jgroups.mping.mcast_port:43366}"
ip_ttl="${jgroups.udp.ip_ttl:2}"
num_initial_members="3"/>
...
...
</config>
192.168.68.201 was replaced by respective IP address at a particular node.
Most likely, multicast traffic is not allow to travel between 192.168.69.x and 192.168.68.x networks which is a common problem when using multicast based PING protocols (MPING). Either, talk to your network guys to allow multicast between those two subnets, or as suggested by Arjun, try using TCPPING specifying the IP addresses of the nodes to connect to. Here's an example of TCPPING configuration.
Try using TCPPING instead of MPING
I am trying to set up zookeeper in replicated node with 3 server.
my config file is like this
tickTime=2000
dataDir=/var/lib/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
I am getting following exception
QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection#642] - Adding vote
2009-09-23 15:30:28,099 - WARN [WorkerSender Thread:QuorumCnxManager#336] -
Cannot open channel to 3 at election address zoo1/172.21.31.159:3888
java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method)> at
sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:
All ports are open and ssh, telnet are also working.
Thanks
Here is a quick checklist:
Do you have a /var/lib/zookeeper/myid file?
Are ids defined in that file in sync with machine names/IPs defined in config (zoo1 having id 1, etc...)?
Not sure why but I had to use 0.0.0.0 as a host name in Zookeeper config for a each of the servers.
I.e:
Server 1:
myid
1
zoo.cfg
server.1=0.0.0.0:2888:3888
server.2=X.X.X.2:2888:3888
server.3=X.X.X.3:2888:3888
Server 2:
myid
2
zoo.cfg
server.1=X.X.X.1:2888:3888
server.2=0.0.0.0:2888:3888
server.3=X.X.X.3:2888:3888
Server 3:
myid
3
zoo.cfg
server.1=X.X.X.1:2888:3888
server.2=X.X.X.2:2888:3888
server.3=0.0.0.0:2888:3888