Issues clustering Artemis on docker - activemq-artemis

Question

I'm trying to get a simple clustering example working on docker of two nodes.
I've used the example broker.xml files from the examples.
Since they are running on the same host machine I've changed the port for the second instance. These ports are exposed and mapped on docker.
However when the instances startup and try to contact each other I get warnings that it cannot connect to the destinations.
node0_1 | 2020-07-11 19:49:10,775 WARN [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge#12be5aac [name=$.artemis.internal.sf.my-cluster.93477bd0-c3af-11ea-bd98-0242ac120003, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.93477bd0-c3af-11ea-bd98-0242ac120003, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=933f3e46-c3af-11ea-a418-0242ac120002], temp=false]#63e7e9f9 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#12be5aac [name=$.artemis.internal.sf.my-cluster.93477bd0-c3af-11ea-bd98-0242ac120003, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.93477bd0-c3af-11ea-bd98-0242ac120003, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=933f3e46-c3af-11ea-a418-0242ac120002], temp=false]#63e7e9f9 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=localhost], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#912966811[nodeUUID=933f3e46-c3af-11ea-a418-0242ac120002, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=localhost, address=, server=ActiveMQServerImpl::serverUUID=933f3e46-c3af-11ea-a418-0242ac120002])) [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=localhost], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying
node1_1 | 2020-07-11 19:49:10,913 WARN [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge#1634c4b9 [name=$.artemis.internal.sf.my-cluster.933f3e46-c3af-11ea-a418-0242ac120002, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.933f3e46-c3af-11ea-a418-0242ac120002, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=93477bd0-c3af-11ea-bd98-0242ac120003], temp=false]#50c16063 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge#1634c4b9 [name=$.artemis.internal.sf.my-cluster.933f3e46-c3af-11ea-a418-0242ac120002, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.933f3e46-c3af-11ea-a418-0242ac120002, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=93477bd0-c3af-11ea-bd98-0242ac120003], temp=false]#50c16063 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=localhost], discoveryGroupConfiguration=null]]::ClusterConnectionImpl#1792384402[nodeUUID=93477bd0-c3af-11ea-bd98-0242ac120003, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=localhost, address=, server=ActiveMQServerImpl::serverUUID=93477bd0-c3af-11ea-bd98-0242ac120003])) [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=localhost], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying
I've tried changing from localhost to IP address of host but this results in a bind error
Failure in initialisation: java.lang.IllegalStateException: AMQ229230: Failed to bind acceptor netty-acceptor to 192.168.0.15:61616
node0_1 | at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.startServerChannels(NettyAcceptor.java:688) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.start(NettyAcceptor.java:456) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl.startAcceptors(RemotingServiceImpl.java:313) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.completeActivation(ActiveMQServerImpl.java:3023) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:78) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:599) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:526) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:70) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:84) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:153) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:101) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:128) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_232]
node0_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_232]
node0_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_232]
node0_1 | at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_232]
node0_1 | at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134) [artemis-boot.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50) [artemis-boot.jar:2.13.0]
node0_1 | Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Cannot assign requested address
I thought it might be something to do with the network interfaces as I have more than one virtual one.
If I set local-bind-address for the broadcast groups and discovery groups I get the following error:
node0_1 | 2020-07-11 20:07:07,774 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: java.lang.IllegalStateException: AMQ219004: Failed to initialise session factory
node0_1 | at org.apache.activemq.artemis.core.server.cluster.ClusterController.configAndAdd(ClusterController.java:204) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.cluster.ClusterController.addClusterConnection(ClusterController.java:169) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.cluster.ClusterManager.deployClusterConnection(ClusterManager.java:612) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.cluster.ClusterManager.deploy(ClusterManager.java:245) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:2854) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:68) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:599) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:526) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:70) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:84) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:153) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:101) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:128) [artemis-cli-2.13.0.jar:2.13.0]
node0_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_232]
node0_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_232]
node0_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_232]
node0_1 | at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_232]
node0_1 | at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134) [artemis-boot.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50) [artemis-boot.jar:2.13.0]
node0_1 | Caused by: ActiveMQInternalErrorException[errorType=INTERNAL_ERROR message=AMQ219004: Failed to initialise session factory]
node0_1 | at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:269) [artemis-core-client-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.server.cluster.ClusterController.configAndAdd(ClusterController.java:202) [artemis-server-2.13.0.jar:2.13.0]
node0_1 | ... 18 more
node0_1 | Caused by: ActiveMQInternalErrorException[errorType=INTERNAL_ERROR message=Cannot assign requested address (Error setting socket option)]
node0_1 | at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.startDiscovery(ServerLocatorImpl.java:283) [artemis-core-client-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:265) [artemis-core-client-2.13.0.jar:2.13.0]
node0_1 | ... 19 more
node0_1 | Caused by: java.net.SocketException: Cannot assign requested address (Error setting socket option)
node0_1 | at java.net.PlainDatagramSocketImpl.socketSetOption0(Native Method) [rt.jar:1.8.0_232]
node0_1 | at java.net.PlainDatagramSocketImpl.socketSetOption(PlainDatagramSocketImpl.java:74) [rt.jar:1.8.0_232]
node0_1 | at java.net.AbstractPlainDatagramSocketImpl.setOption(AbstractPlainDatagramSocketImpl.java:309) [rt.jar:1.8.0_232]
node0_1 | at java.net.MulticastSocket.setInterface(MulticastSocket.java:471) [rt.jar:1.8.0_232]
node0_1 | at org.apache.activemq.artemis.api.core.UDPBroadcastEndpointFactory$UDPBroadcastEndpoint.openClient(UDPBroadcastEndpointFactory.java:214) [artemis-core-client-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.cluster.DiscoveryGroup.start(DiscoveryGroup.java:111) [artemis-core-client-2.13.0.jar:2.13.0]
node0_1 | at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.startDiscovery(ServerLocatorImpl.java:281) [artemis-core-client-2.13.0.jar:2.13.0]
node0_1 | ... 20 more
That being said I'm not sure I'm setting this value correctly. If I have two network interfaces, one with a 172.22.0.x for the VMs and the other a 192.168.0.x for the host machine. What value should I use?
broxer.xml 1
<?xml version="1.0"?>
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<!-- from 1.0.0 to 1.5.5 the following line should be : <core xmlns="urn:activemq:core"> -->
<core xmlns="urn:activemq:core" xsi:schemaLocation="urn:activemq:core ">
<bindings-directory>./data/bindings</bindings-directory>
<journal-directory>./data/journal</journal-directory>
<large-messages-directory>./data/largemessages</large-messages-directory>
<paging-directory>./data/paging</paging-directory>
<!-- Connectors -->
<connectors>
<connector name="netty-connector">tcp://localhost:61616</connector>
</connectors>
<!-- Acceptors -->
<acceptors>
<acceptor name="netty-acceptor">tcp://localhost:61616</acceptor>
</acceptors>
<!-- Clustering configuration -->
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<group-address>${udp-address:231.7.7.7}</group-address>
<group-port>9876</group-port>
<broadcast-period>100</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<group-address>${udp-address:231.7.7.7}</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<!-- Other config -->
<security-settings>
<!--security for example queue-->
<security-setting match="exampleQueue">
<permission roles="guest" type="createDurableQueue"/>
<permission roles="guest" type="deleteDurableQueue"/>
<permission roles="guest" type="createNonDurableQueue"/>
<permission roles="guest" type="deleteNonDurableQueue"/>
<permission roles="guest" type="consume"/>
<permission roles="guest" type="send"/>
</security-setting>
</security-settings>
<addresses>
<address name="exampleQueue">
<anycast>
<queue name="exampleQueue"/>
</anycast>
</address>
</addresses>
</core>
</configuration>
broxer.xml 2
<?xml version="1.0"?>
<configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<!-- from 1.0.0 to 1.5.5 the following line should be : <core xmlns="urn:activemq:core"> -->
<core xmlns="urn:activemq:core" xsi:schemaLocation="urn:activemq:core ">
<bindings-directory>./data/bindings</bindings-directory>
<journal-directory>./data/journal</journal-directory>
<large-messages-directory>./data/largemessages</large-messages-directory>
<paging-directory>./data/paging</paging-directory>
<!-- Connectors -->
<connectors>
<connector name="netty-connector">tcp://localhost:61617</connector>
</connectors>
<!-- Acceptors -->
<acceptors>
<acceptor name="netty-acceptor">tcp://localhost:61617</acceptor>
</acceptors>
<!-- Clustering configuration -->
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<group-address>${udp-address:231.7.7.7}</group-address>
<group-port>9876</group-port>
<broadcast-period>100</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<group-address>${udp-address:231.7.7.7}</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<!-- Other config -->
<security-settings>
<!--security for example queue-->
<security-setting match="exampleQueue">
<permission roles="guest" type="createDurableQueue"/>
<permission roles="guest" type="deleteDurableQueue"/>
<permission roles="guest" type="createNonDurableQueue"/>
<permission roles="guest" type="deleteNonDurableQueue"/>
<permission roles="guest" type="consume"/>
<permission roles="guest" type="send"/>
</security-setting>
</security-settings>
<addresses>
<address name="exampleQueue">
<anycast>
<queue name="exampleQueue"/>
</anycast>
</address>
</addresses>
</core>
</configuration>
docker-compose file
version: '3.3'
services:
node0:
image: vromero/activemq-artemis:latest
volumes:
- ./node0/etc/:/var/lib/artemis/etc
environment:
ENABLE_JMX_EXPORTER: "true"
ARTEMIS_MIN_MEMORY: "1512M"
ARTEMIS_MAX_MEMORY: "2000M"
ARTEMIS_USERNAME: "artemis"
ARTEMIS_PASSWORD: "password"
ports:
- "8161:8161"
- "61616:61616"
node1:
image: vromero/activemq-artemis:latest
volumes:
- ./node1/etc/:/var/lib/artemis/etc
environment:
ENABLE_JMX_EXPORTER: "true"
ARTEMIS_MIN_MEMORY: "1512M"
ARTEMIS_MAX_MEMORY: "2000M"
ARTEMIS_USERNAME: "artemis"
ARTEMIS_PASSWORD: "password"
ports:
- "8162:8161"
- "61617:61617"

Answer 1

Network bridge is the default network driver used by docker and creates a virtual interface in the container that’s not available on the host but allows containers connected to the same bridge network to communicate.
The IP address of the broker used in broker.xml for the connector and the acceptor should match with the IP address of the container virtual network interface. This IP address is assigned dynamically but docker-compose allows to define a custom network with a fixed addresses, ie:
networks:
artemis-cluster-network:
ipam:
config:
- subnet: 172.20.0.0/16
The UDP multicast with the address 231.7.7.7 should work and you can check it with the following docker-compose.yml:
version: '3.0'
services:
udp-receiver:
image: alpine/socat
entrypoint: ''
command: sh -c "hostname -i ; socat UDP4-RECVFROM:6666,ip-add-membership=231.7.7.7:0.0.0.0 STDIO"
networks:
artemis-cluster-network:
ipv4_address: 172.20.0.100
udp-sender:
image: alpine/socat
depends_on:
- udp-receiver
entrypoint: ''
command: sh -c "hostname -i ; echo 'Hello World' | socat - UDP4-DATAGRAM:231.7.7.7:6666,so-broadcast"
networks:
artemis-cluster-network:
ipv4_address: 172.20.0.101
networks:
artemis-cluster-network:
ipam:
config:
- subnet: 172.20.0.0/16