ceph-deploy mon add node2 admin_socket: exception - ceph

Question

Follow the official quick-ceph-deploy, at the adding-monitor step, meet the following problem:
[ubuntu#admin-node my-cluster]$ ceph-deploy mon add node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.37): /usr/bin/ceph-deploy mon add node2
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : add
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1b0fc20>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] mon : ['node2']
[ceph_deploy.cli][INFO ] func : <function mon at 0x1b07320>
[ceph_deploy.cli][INFO ] address : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mon][INFO ] ensuring configuration of new mon host: node2
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node2
[node2][DEBUG ] connection detected need for sudo
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.mon][DEBUG ] Adding mon to cluster ceph, host node2
[ceph_deploy.mon][DEBUG ] using mon address by resolving host: 172.24.2.232
[ceph_deploy.mon][DEBUG ] detecting platform for host node2 ...
[node2][DEBUG ] connection detected need for sudo
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO ] distro info: CentOS Linux 7.3.1611 Core
[node2][DEBUG ] determining if provided host has same hostname in remote
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] adding mon to node2
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][DEBUG ] create the mon path if it does not exist
[node2][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node2/done
[node2][DEBUG ] create a done file to avoid re-doing the mon deployment
[node2][DEBUG ] create the init path if it does not exist
[node2][INFO ] Running command: sudo systemctl enable ceph.target
[node2][INFO ] Running command: sudo systemctl enable ceph-mon#node2
[node2][INFO ] Running command: sudo systemctl start ceph-mon#node2
[node2][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
[node2][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[node2][WARNIN] node2 is not defined in `mon initial members`
[node2][WARNIN] monitor node2 does not exist in monmap
[node2][WARNIN] neither `public_addr` nor `public_network` keys are defined for monitors
[node2][WARNIN] monitors may not be able to form quorum
[node2][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
[node2][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[node2][WARNIN] monitor: mon.node2, might not be running yet
ceph.conf:
[global]
fsid = 5ec213d4-ae42-44c2-81d1-d7bdbee7f36a
mon_initial_members = node1
mon_host = 172.24.2.230
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
# by honghe
osd pool default size = 2
the ceph status is:
[ubuntu#admin-node my-cluster]$ ceph status
cluster 5ec213d4-ae42-44c2-81d1-d7bdbee7f36a
health HEALTH_OK
monmap e1: 1 mons at {node1=172.24.2.230:6789/0}
election epoch 3, quorum 0 node1
osdmap e25: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v12180: 112 pgs, 7 pools, 1588 bytes data, 171 objects
19243 MB used, 90095 MB / 106 GB avail
112 active+clean
[ubuntu#admin-node my-cluster]$ ceph -v
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
Any help will be appreciated!

Answer 1

Resolved.
I miss the configuration of newtork.
cause nodeX have 2 IF:
[ubuntu#node1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:a8:33:13 brd ff:ff:ff:ff:ff:ff
inet 172.24.2.230/24 brd 172.24.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::9aae:a37b:289d:1745/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:7f:98:49 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.210/24 brd 192.168.122.255 scope global dynamic eth1
valid_lft 2754sec preferred_lft 2754sec
inet6 fe80::19b4:5768:7469:5767/64 scope link
valid_lft forever preferred_lft forever
So we have to config the network
If you have more than one network interface, add the public network setting under the [global] section of your Ceph configuration file. See the Network Configuration Reference for details.
Just as follow:
[global]
fsid = 5ec213d4-ae42-44c2-81d1-d7bdbee7f36a
mon_initial_members = node1
mon_host = 172.24.2.230
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
# by honghe
osd pool default size = 2
public network = 172.24.2.0/24
More over, add configuration of cluster network to use another IF can get better performance.