I am deploying Mesos/Marathon/zookeeper on a cluster of one physical machine : is this a viable configuration ?
(I have two machines, but begin by the first one)
When launching Mesos I get the following result on the page : "No master is currently leading, This master is not a leader ..."
I put the quorum at 1, is it a possible value ? I tried 0 but mesos doesn't even start.
EDIT
cat mesos-master.INFO
Log file created at: 2015/10/19 15:58:15
Running on machine: 192.168.0.38
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1019 15:58:15.592771 8 logging.cpp:172] INFO level logging started!
I1019 15:58:15.593093 8 main.cpp:229] Build: 2015-10-12 20:57:28 by root
I1019 15:58:15.593111 8 main.cpp:231] Version: 0.25.0
I1019 15:58:15.593123 8 main.cpp:234] Git tag: 0.25.0
I1019 15:58:15.593135 8 main.cpp:238] Git SHA: 2dd7f7ee115fe00b8e098b0a10762a4fa8f4600f
I1019 15:58:15.593276 8 main.cpp:252] Using 'HierarchicalDRF' allocator
I1019 15:58:15.660604 8 leveldb.cpp:176] Opened db in 67.183194ms
I1019 15:58:15.678915 8 leveldb.cpp:183] Compacted db in 18.242065ms
I1019 15:58:15.678962 8 leveldb.cpp:198] Created db iterator in 14924ns
I1019 15:58:15.678982 8 leveldb.cpp:204] Seeked to beginning of db in 1323ns
I1019 15:58:15.678998 8 leveldb.cpp:273] Iterated through 0 keys in the db in 2556ns
I1019 15:58:15.679056 8 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I1019 15:58:15.680054 30 log.cpp:238] Attempting to join replica to ZooKeeper group
I1019 15:58:15.680531 36 recover.cpp:449] Starting replica recovery
I1019 15:58:15.680735 36 recover.cpp:475] Replica is in EMPTY status
I1019 15:58:15.683269 8 main.cpp:465] Starting Mesos master
I1019 15:58:15.684293 35 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request
I1019 15:58:15.684648 37 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1019 15:58:15.685711 31 recover.cpp:566] Updating replica status to STARTING
I1019 15:58:15.688724 31 master.cpp:376] Master 74ee40e5-16b6-4a40-8288-4f563806b5cb (192.168.0.38) started on 192.168.0.38:5050
I1019 15:58:15.688765 31 master.cpp:378] Flags at startup: --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="false" --authenticators="crammd5" --authorizers="local" --cluster="" --framework_sorter="drf" --help="false" --hostname="192.168.0.38" --hostname_lookup="true" --initialize_driver_logging="true" --ip="192.168.0.38" --log_auto_initialize="true" --log_dir="/etc/mesos/logs" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --port="5050" --quiet="false" --quorum="1" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/usr/share/mesos/webui" --work_dir="/var/lib/mesos" --zk="zk://192.168.0.38:2181/mesos" --zk_session_timeout="10secs"
I1019 15:58:15.689028 31 master.cpp:425] Master allowing unauthenticated frameworks to register
I1019 15:58:15.689049 31 master.cpp:430] Master allowing unauthenticated slaves to register
I1019 15:58:15.689071 31 master.cpp:467] Using default 'crammd5' authenticator
W1019 15:58:15.689100 31 authenticator.cpp:505] No credentials provided, authentication requests will be refused
I1019 15:58:15.689422 31 authenticator.cpp:512] Initializing server SASL
I1019 15:58:15.695821 31 contender.cpp:149] Joining the ZK group
I1019 15:58:15.699548 34 group.cpp:331] Group process (group(2)#192.168.0.38:5050) connected to ZooKeeper
I1019 15:58:15.706737 30 master.cpp:1542] Successfully attached file '/etc/mesos/logs/mesos-master.INFO'
I1019 15:58:15.706755 34 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I1019 15:58:15.706826 34 group.cpp:403] Trying to create path '/mesos/log_replicas' in ZooKeeper
I1019 15:58:15.710538 35 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 24.699241ms
I1019 15:58:15.710598 35 replica.cpp:323] Persisted replica status to STARTING
I1019 15:58:15.710695 37 recover.cpp:475] Replica is in STARTING status
I1019 15:58:15.710979 32 replica.cpp:641] Replica in STARTING status received a broadcasted recover request
I1019 15:58:15.711148 31 recover.cpp:195] Received a recover response from a replica in STARTING status
I1019 15:58:15.711293 37 recover.cpp:566] Updating replica status to VOTING
I1019 15:58:15.723206 37 group.cpp:331] Group process (group(1)#192.168.0.38:5050) connected to ZooKeeper
I1019 15:58:15.730281 37 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1019 15:58:15.730325 37 group.cpp:403] Trying to create path '/mesos/log_replicas' in ZooKeeper
I1019 15:58:15.731256 33 group.cpp:331] Group process (group(4)#192.168.0.38:5050) connected to ZooKeeper
I1019 15:58:15.731343 33 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1019 15:58:15.731359 33 group.cpp:403] Trying to create path '/mesos' in ZooKeeper
I1019 15:58:15.731947 30 group.cpp:331] Group process (group(3)#192.168.0.38:5050) connected to ZooKeeper
I1019 15:58:15.733675 30 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I1019 15:58:15.734716 30 group.cpp:403] Trying to create path '/mesos' in ZooKeeper
I1019 15:58:15.734612 31 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 23.245997ms
I1019 15:58:15.734902 31 replica.cpp:323] Persisted replica status to VOTING
I1019 15:58:15.734987 32 recover.cpp:580] Successfully joined the Paxos group
I1019 15:58:15.735080 32 recover.cpp:464] Recover process terminated
I1019 15:58:15.745573 33 network.hpp:415] ZooKeeper group memberships changed
I1019 15:58:15.745687 32 group.cpp:674] Trying to get '/mesos/log_replicas/0000000030' in ZooKeeper
I1019 15:58:15.749068 35 network.hpp:463] ZooKeeper group PIDs: { log-replica(1)#192.168.0.38:5050 }
I1019 15:58:15.750468 36 contender.cpp:265] New candidate (id='30') has entered the contest for leadership
I1019 15:58:15.751054 33 detector.cpp:156] Detected a new leader: (id='30')
I1019 15:58:15.751231 34 group.cpp:674] Trying to get '/mesos/json.info_0000000030' in ZooKeeper
I1019 15:58:15.751909 33 detector.cpp:481] A new leading master (UPID=master#192.168.0.38:5050) is detected
I1019 15:58:15.752105 34 master.cpp:1603] The newly elected leader is master#192.168.0.38:5050 with id 74ee40e5-16b6-4a40-8288-4f563806b5cb
I1019 15:58:15.752182 34 master.cpp:1616] Elected as the leading master!
I1019 15:58:15.752208 34 master.cpp:1376] Recovering from registrar
I1019 15:58:15.752290 33 registrar.cpp:309] Recovering registrar
I1019 15:58:15.752581 35 log.cpp:661] Attempting to start the writer
I1019 15:58:15.753067 34 replica.cpp:477] Replica received implicit promise request with proposal 1
I1019 15:58:15.774773 34 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 21.627305ms
I1019 15:58:15.774910 34 replica.cpp:345] Persisted promised to 1
I1019 15:58:15.775125 34 coordinator.cpp:231] Coordinator attemping to fill missing position
I1019 15:58:15.775501 33 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2
I1019 15:58:15.790747 33 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 15.185008ms
I1019 15:58:15.790833 33 replica.cpp:679] Persisted action at 0
I1019 15:58:15.791260 33 replica.cpp:511] Replica received write request for position 0
I1019 15:58:15.791342 33 leveldb.cpp:438] Reading position from leveldb took 23444ns
I1019 15:58:15.803988 33 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 12.606014ms
I1019 15:58:15.804051 33 replica.cpp:679] Persisted action at 0
I1019 15:58:15.804256 32 replica.cpp:658] Replica received learned notice for position 0
I1019 15:58:15.815990 32 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 11.675285ms
I1019 15:58:15.816064 32 replica.cpp:679] Persisted action at 0
I1019 15:58:15.816087 32 replica.cpp:664] Replica learned NOP action at position 0
I1019 15:58:15.816222 34 log.cpp:677] Writer started with ending position 0
I1019 15:58:15.816537 32 leveldb.cpp:438] Reading position from leveldb took 9246ns
I1019 15:58:15.817867 36 registrar.cpp:342] Successfully fetched the registry (0B) in 65.515008ms
I1019 15:58:15.817951 36 registrar.cpp:441] Applied 1 operations in 11601ns; attempting to update the 'registry'
I1019 15:58:15.819144 30 log.cpp:685] Attempting to append 173 bytes to the log
I1019 15:58:15.819236 32 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 1
I1019 15:58:15.819448 30 replica.cpp:511] Replica received write request for position 1
I1019 15:58:15.832018 30 leveldb.cpp:343] Persisting action (192 bytes) to leveldb took 12.520293ms
I1019 15:58:15.832092 30 replica.cpp:679] Persisted action at 1
I1019 15:58:15.832268 35 replica.cpp:658] Replica received learned notice for position 1
I1019 15:58:15.844065 35 leveldb.cpp:343] Persisting action (194 bytes) to leveldb took 11.769077ms
I1019 15:58:15.844130 35 replica.cpp:679] Persisted action at 1
I1019 15:58:15.844175 35 replica.cpp:664] Replica learned APPEND action at position 1
I1019 15:58:15.844462 31 registrar.cpp:486] Successfully updated the 'registry' in 26.468864ms
I1019 15:58:15.844506 30 log.cpp:704] Attempting to truncate the log to 1
I1019 15:58:15.844571 31 registrar.cpp:372] Successfully recovered registrar
I1019 15:58:15.844599 30 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 2
I1019 15:58:15.844714 31 master.cpp:1413] Recovered 0 slaves from the Registry (134B) ; allowing 10mins for slaves to re-register
I1019 15:58:15.844790 37 replica.cpp:511] Replica received write request for position 2
I1019 15:58:15.858352 37 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 13.469108ms
I1019 15:58:15.858502 37 replica.cpp:679] Persisted action at 2
I1019 15:58:15.858723 37 replica.cpp:658] Replica received learned notice for position 2
I1019 15:58:15.874608 37 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 15.810315ms
I1019 15:58:15.874725 37 leveldb.cpp:401] Deleting ~1 keys from leveldb took 27159ns
I1019 15:58:15.874747 37 replica.cpp:679] Persisted action at 2
I1019 15:58:15.874764 37 replica.cpp:664] Replica learned TRUNCATE action at position 2
I1019 15:58:19.851761 37 master.cpp:2179] Received SUBSCRIBE call for framework 'marathon' at scheduler-45acfb6e-2a61-46e8-bef3-7bc5d0e76567#192.168.0.38:33773
I1019 15:58:19.851974 37 master.cpp:2250] Subscribing framework marathon with checkpointing enabled and capabilities [ ]
I1019 15:58:19.852358 30 hierarchical.hpp:515] Added framework a927b696-0597-4762-9969-f1fe2a5d7a2e-0000
I1019 15:58:34.856995 34 master.cpp:4640] Performing implicit task state reconciliation for framework a927b696-0597-4762-9969-f1fe2a5d7a2e-0000 (marathon) at scheduler-45acfb6e-2a61-46e8-bef3-7bc5d0e76567#192.168.0.38:33773
w3m http://192.168.0.38:5050
[loading]
Toggle navigation Mesos
• Frameworks
• Slaves
• Offers
• {{state.cluster}}
No master is currently leading ...
× This master is not the leader, redirecting in {{redirect / 1000}} seconds ... go now
{{ alert.title }}
{{ alert.message }}
• {{ bullet }}
Related
we have 3 Kafka brokers on Linux RHEL 7.6 ( 3 linux machines )
kafka version is 2.7.X
brokers ID's are - 1010,1011,1012
from kafka described we can see the following
Topic: __consumer_offsets Partition: 0 Leader: none Replicas: 1011,1010,1012 Isr: 1010
Topic: __consumer_offsets Partition: 1 Leader: 1012 Replicas: 1012,1011,1010 Isr: 1012,1011
Topic: __consumer_offsets Partition: 2 Leader: 1011 Replicas: 1010,1012,1011 Isr: 1011,1012
Topic: __consumer_offsets Partition: 3 Leader: none Replicas: 1011,1012,1010 Isr: 1010
Topic: __consumer_offsets Partition: 4 Leader: 1011 Replicas: 1012,1010,1011 Isr: 1011
Topic: __consumer_offsets Partition: 5 Leader: none Replicas: 1010,1011,1012 Isr: 1010
from Zookeeper cli we can see that broker id 1010 not defined
[zk: localhost:2181(CONNECTED) 10] ls /brokers/ids
[1011, 1012]
and from the log - state-change.log
we can see the following
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-6 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-9 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-8 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-11 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-10 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-46 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-45 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-48 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-47 as the local replica for the partition is in an offline log directory (state.change.logger)
[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-49 as the local replica for the partition is in an offline log directory (state.change.logger)
by ls -ltr , we can see that controller.log and state-change.log are not update from Dec 16
-rwxr-xr-x 1 root kafka 343477146 Dec 16 14:15 controller.log
-rwxr-xr-x 1 root kafka 207911766 Dec 16 14:15 state-change.log
-rw-r--r-- 1 root kafka 68759461 Dec 16 14:15 kafkaServer-gc.log.6.current
-rwxr-xr-x 1 root kafka 6570543 Dec 17 09:42 log-cleaner.log
-rw-r--r-- 1 root kafka 524288242 Dec 20 00:39 server.log.10
-rw-r--r-- 1 root kafka 524289332 Dec 20 01:37 server.log.9
-rw-r--r-- 1 root kafka 524288452 Dec 20 02:35 server.log.8
-rw-r--r-- 1 root kafka 524288625 Dec 20 03:33 server.log.7
-rw-r--r-- 1 root kafka 524288395 Dec 20 04:30 server.log.6
-rw-r--r-- 1 root kafka 524288237 Dec 20 05:27 server.log.5
-rw-r--r-- 1 root kafka 524289136 Dec 20 06:25 server.log.4
-rw-r--r-- 1 root kafka 524288142 Dec 20 07:25 server.log.3
-rw-r--r-- 1 root kafka 524288187 Dec 20 08:21 server.log.2
-rw-r--r-- 1 root kafka 524288094 Dec 20 10:52 server.log.1
-rw-r--r-- 1 root kafka 323361 Dec 20 19:50 kafkaServer-gc.log.0.current
-rw-r--r-- 1 root kafka 323132219 Dec 20 19:50 server.log
-rwxr-xr-x 1 root kafka 15669106 Dec 20 19:50 kafkaServer.out
what we did until now is that:
we restart all 3 zookeeper servers
we restart all kafka brokers
but still kafka broker 1010 appears as leader none , and not in zookeeper data
additional info
[zk: localhost:2181(CONNECTED) 11] get /controller
{"version":1,"brokerid":1011,"timestamp":"1640003679634"}
cZxid = 0x4900000b0c
ctime = Mon Dec 20 12:34:39 UTC 2021
mZxid = 0x4900000b0c
mtime = Mon Dec 20 12:34:39 UTC 2021
pZxid = 0x4900000b0c
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x27dd7cf43350080
dataLength = 57
numChildren = 0
from kafka01
more meta.properties
#
#Tue Nov 16 07:45:36 UTC 2021
cluster.id=D3KpekCETmaNveBJzE6PZg
version=0
broker.id=1010
relevant ideas
in topics disk we have the following files ( additionally to topics partitions )
-rw-r--r-- 1 root kafka 91 Nov 16 07:45 meta.properties
-rw-r--r-- 1 root kafka 161 Dec 15 16:04 cleaner-offset-checkpoint
-rw-r--r-- 1 root kafka 13010 Dec 15 16:20 replication-offset-checkpoint
-rw-r--r-- 1 root kafka 1928 Dec 17 09:42 recovery-point-offset-checkpoint
-rw-r--r-- 1 root kafka 80 Dec 17 09:42 log-start-offset-checkpoint
any idea if deletion of one or more of above files can help with our issue?
All that you've shown is that broker 1010 isn't healthy and you probably have unclear leader election disabled.
ls /brokers/ids shows you the running, healthy brokers from Zookeeper's point of view.
Meanwhile, the data in the /topics znodes refers to a broker listed in the replica set that is not running, or at least not reporting back to Zookeeper, which you'd see in server.log
If you have another broker, you can use the partition reassignment tool to manually remove/change broker 1010 data from each partition of all topics it hosts, which would remove old replica information in Zookeeper, and should force a new leader
You shouldn't delete checkpoint files, but you can delete old, rotated log files after you've determined they're not needed anymore
i have this situation and cand access to ceph dashboard.i haad 5 mon but 2 of them went down and one of them is the bootstrap mon node so that have mgr and I got this from that node.
2020-10-14T18:59:46.904+0330 7f9d2e8e9700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
cluster:
id: e97c1944-e132-11ea-9bdd-e83935b1c392
health: HEALTH_WARN
no active mgr
services:
mon: 3 daemons, quorum srv4,srv5,srv6 (age 2d)
mgr: no daemons active (since 2d)
mds: heyatfs:1 {0=heyfs.srv10.lxizhc=up:active} 1 up:standby
osd: 54 osds: 54 up (since 47h), 54 in (since 3w)
task status:
scrub status:
mds.heyfs.srv10.lxizhc: idle
data:
pools: 3 pools, 65 pgs
objects: 223.95k objects, 386 GiB
usage: 1.2 TiB used, 97 TiB / 98 TiB avail
pgs: 65 active+clean
io:
client: 105 KiB/s rd, 328 KiB/s wr, 0 op/s rd, 0 op/s wr
I have to say the whole story, I used cephadm to create my cluster at first and I'm so new to ceph i have 15 servers and 14 of them have OSD container and 5 of them had mon and my bootstrap mon that is srv2 have mgr.
2 of these servers have public IP and I used one of them as a client (I know this structure have a lot of question in it but my company forces me to do it and also I'm new to ceph so it's how it's now). 2 weeks ago I lost 2 OSD and I said to datacenter who gives me these servers to change that 2 HDD they restart those servers and unfortunately, those servers were my Mon server. after they restarted those servers on of them came back srv5 but I could see srv3 is out of quorum
so i begon to solve this problem so I used this command in ceph shell --fsid ...
ceph orch apply mon srv3
ceph mon remove srv3
after some while I see in my dashboard srv2 my boostrap mon and mgr is not working and when I used ceph -s ssrv2 isn't there and I can see srv2 mon in removed directory
root#srv2:/var/lib/ceph/e97c1944-e132-11ea-9bdd-e83935b1c392# ls
crash crash.srv2 home mgr.srv2.xpntaf osd.0 osd.1 osd.2 osd.3 removed
but mgr.srv2.xpntaf is running and unfortunately, I lost my access to ceph dashboard now
i tried to add srv2 and 3 to monmap with
576 ceph orch daemon add mon srv2:172.32.X.3
577 history | grep dump
578 ceph mon dump
579 ceph -s
580 ceph mon dump
581 ceph mon add srv3 172.32.X.4:6789
and now
root#srv2:/# ceph -s
cluster:
id: e97c1944-e132-11ea-9bdd-e83935b1c392
health: HEALTH_WARN
no active mgr
2/5 mons down, quorum srv4,srv5,srv6
services:
mon: 5 daemons, quorum srv4,srv5,srv6 (age 16h), out of quorum: srv2, srv3
mgr: no daemons active (since 2d)
mds: heyatfs:1 {0=heyatfs.srv10.lxizhc=up:active} 1 up:standby
osd: 54 osds: 54 up (since 2d), 54 in (since 3w)
task status:
scrub status:
mds.heyatfs.srv10.lxizhc: idle
data:
pools: 3 pools, 65 pgs
objects: 223.95k objects, 386 GiB
usage: 1.2 TiB used, 97 TiB / 98 TiB avail
pgs: 65 active+clean
io:
client: 105 KiB/s rd, 328 KiB/s wr, 0 op/s rd, 0 op/s wr
and I must say ceph orch host ls doesn't work and it hangs when I run it and I think it's because of that err no active mgr and also when I see that removed directory mon.srv2 is there and you can see unit.run file so I used that command to run the container again but it says mon.srv2 isn't on mon map and doesn't have specific IP and by the way I must say after ceph orch apply mon srv3 i could see a new container with a new fsid in srv3 server
I now my whole problem is because I ran this command ceph orch apply mon srv3
because when you see the installation document :
To deploy monitors on a specific set of hosts:
# ceph orch apply mon *<host1,host2,host3,...>*
Be sure to include the first (bootstrap) host in this list.
and I didn't see that line !!!
now I manage to have another mgr running but I got this
root#srv2:/var/lib/ceph/mgr# ceph -s
2020-10-15T13:11:59.080+0000 7f957e9cd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
cluster:
id: e97c1944-e132-11ea-9bdd-e83935b1c392
health: HEALTH_ERR
1 stray daemons(s) not managed by cephadm
2 mgr modules have failed
2/5 mons down, quorum srv4,srv5,srv6
services:
mon: 5 daemons, quorum srv4,srv5,srv6 (age 20h), out of quorum: srv2, srv3
mgr: srv4(active, since 8m)
mds: heyatfs:1 {0=heyatfs.srv10.lxizhc=up:active} 1 up:standby
osd: 54 osds: 54 up (since 2d), 54 in (since 3w)
task status:
scrub status:
mds.heyatfs.srv10.lxizhc: idle
data:
pools: 3 pools, 65 pgs
objects: 301.77k objects, 537 GiB
usage: 1.6 TiB used, 97 TiB / 98 TiB avail
pgs: 65 active+clean
io:
client: 180 KiB/s rd, 597 B/s wr, 0 op/s rd, 0 op/s wr
and when I run the ceph orch host ls i see this
root#srv2:/var/lib/ceph/mgr# ceph orch host ls
HOST ADDR LABELS STATUS
srv10 172.32.x.11
srv11 172.32.x.12
srv12 172.32.x.13
srv13 172.32.x.14
srv14 172.32.x.15
srv15 172.32.x.16
srv2 srv2
srv3 172.32.x.4
srv4 172.32.x.5
srv5 172.32.x.6
srv6 172.32.x.7
srv7 172.32.x.8
srv8 172.32.x.9
srv9 172.32.x.10
usually after kafka cluster scratch installation I saw this files under /data/kafka-logs ( kafka broker logs. where all topics should be located )
ls -ltr
-rw-r--r-- 1 kafka hadoop 0 Jan 9 10:07 cleaner-offset-checkpoint
-rw-r--r-- 1 kafka hadoop 57 Jan 9 10:07 meta.properties
drwxr-xr-x 2 kafka hadoop 4096 Jan 9 10:51 _schemas-0
-rw-r--r-- 1 kafka hadoop 17 Jan 10 07:39 recovery-point-offset-checkpoint
-rw-r--r-- 1 kafka hadoop 17 Jan 10 07:39 replication-offset-checkpoint
but on some other Kafka scratch installation we saw the folder - /data/kafka-logs is empty
is this indicate on problem ?
note - we still not create the topics
I'm not sure when each checkpoint file is created (though, they track log cleaner and replication offsets), but I assume that the meta properties is created at broker startup.
Otherwise, you would see one folder per Topic-partition, for example, looks like you had one topic created, _schemas.
If you only see one partition folder out of multiple brokers, then your replication factor for that topic is set to 1
Using Apache Kafka with a single node (1 Zookeeper, 1 Broker) I get this exception (repeated multiple times):
kafka.common.NoReplicaOnlineException: No replica in ISR for partition __consumer_offsets-2 is alive. Live brokers are: [Set()], ISR brokers are: [0]
What does it mean? Note, I am starting the KafkaServer programmatically, and I am able to send and consume from a topic using the CLI tools.
It seems I should tell this node that it is operation in standalone mode - how should I do this?
This seems to happen during startup.
Full exception:
17-11-07 19:43:44 NP-3255AJ193091.home ERROR [state.change.logger:107]
- [Controller id=0 epoch=54] Initiated state change for partition __consumer_offsets-16 from OfflinePartition to OnlinePartition failed
kafka.utils.ShutdownableThread.run ShutdownableThread.scala:
64
kafka.controller.ControllerEventManager$ControllerEventThread.doWork
ControllerEventManager.scala: 52
kafka.metrics.KafkaTimer.time KafkaTimer.scala: 31
kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply
ControllerEventManager.scala: 53 (repeats 2 times)
kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp
ControllerEventManager.scala: 53
kafka.controller.KafkaController$Startup$.process
KafkaController.scala: 1581
kafka.controller.KafkaController.elect KafkaController.scala:
1681
kafka.controller.KafkaController.onControllerFailover
KafkaController.scala: 298
kafka.controller.PartitionStateMachine.startup
PartitionStateMachine.scala: 58
kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange
PartitionStateMachine.scala: 81
scala.collection.TraversableLike$WithFilter.foreach
TraversableLike.scala: 732
scala.collection.mutable.HashMap.foreach
HashMap.scala: 130
scala.collection.mutable.HashMap.foreachEntry
HashMap.scala: 40
scala.collection.mutable.HashTable$class.foreachEntry
HashTable.scala: 236
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply
HashMap.scala: 130 (repeats 2 times)
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply
TraversableLike.scala: 733
kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply
PartitionStateMachine.scala: 81
kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply
PartitionStateMachine.scala: 84
kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange
PartitionStateMachine.scala: 163
kafka.controller.PartitionStateMachine.electLeaderForPartition
PartitionStateMachine.scala: 303
kafka.controller.OfflinePartitionLeaderSelector.selectLeader
PartitionLeaderSelector.scala: 65
kafka.common.NoReplicaOnlineException: No replica in ISR for partition
__consumer_offsets-16 is alive. Live brokers are: [Set()], ISR brokers are: [0]
I have a two node Kafka cluster with 48 gb disk allotted to each.
The server.properties is set to retain logs upto 48 hours or log segments up to 1 GB. Here it is :
log.retention.hours=48
log.retention.bytes=1073741824
log.segment.bytes=1073741824
I have 30 partitons for a topic. Here are the disk usage stats for one of these partitions:
-rw-r--r-- 1 root root 1.9M Apr 14 00:06 00000000000000000000.index
-rw-r--r-- 1 root root 1.0G Apr 14 00:06 00000000000000000000.log
-rw-r--r-- 1 root root 0 Apr 14 00:06 00000000000000000000.timeindex
-rw-r--r-- 1 root root 10M Apr 14 12:43 00000000000001486744.index
-rw-r--r-- 1 root root 73M Apr 14 12:43 00000000000001486744.log
-rw-r--r-- 1 root root 10M Apr 14 00:06 00000000000001486744.timeindex
As you can clearly see, we have a log segment of 1 gb. But as per my understanding, it should have already been deleted. Also, its been more than 48 hours since these logs were rolled out by Kafka. Thoughts?
In your case, you set log.retention.bytes and log.segment.bytes to the same value, which means there is always no candidate of deletable segment, so no delete happens.
The algorithm is:
firstly calculate the difference. In your case, the difference is 73MB (73MB + 1GB - 1GB)
Iterator all the non-active log segments, compare its size with the diff
If diff > log segment size, mark this segment deletable, and decrement the diff by the size
Otherwise, mark this segment undeletable and try with the next log segment.
Answering my own question:
Let's say that log.retention.hours has value 24 hours and log.retention.bytes and log.segment.bytes are both set to 1 GB. When the value of the log reaches 1 GB (call this Old Log), a new log segment is created (call this New Log). The Old Log is then deleted 24 hours after the New Log is created.
In my case, the New Log was created about 25 hours before I posted this question. I dynamically changed the retention.ms value for a topic (which is maintained by Zookeeper, and not the Kafka cluster, and therefore does not require Kafka restart) to 24 hours, my old logs were immediately deleted.