ceph cluster break down after power failure, how can I save my data? - ceph

Question

I have a 4 node ceph cluster deployed with cephadm on rocky9.0, ceph version 17.2.5. podman version 4.1.1
The entire cluster was broke down after an unexpected power failure.
cluster conf
# minimal ceph.conf for bf74b922-5d17-11ed-977a-525400b60738
[global]
fsid = bf74b922-5d17-11ed-977a-525400b60738
mon_host = [v2:192.168.129.190:3300/0,v1:192.168.129.190:6789/0] [v2:192.168.129.198:3300/0,v1:192.168.129.198:6789/0] [v2:192.168.129.232:3300/0,v1:192.168.129.232:6789/0]
[mon.ceph01]
public network = 192.168.129.0/24
check cluster status
ceph -s time out error message:
2023-02-02T08:06:05.037+0000 7f663699d700 0 monclient(hunting): authenticate timed out after 300
check osd service
No output running podman logs ceph-bf74b922-5d17-11ed-977a-525400b60738-osd-2
dashboard available
also checked :
hard drive, all with good health.
selinux and firewalld, disabled
podman, startus ok
network, status ok
DNS, status ok
NTP, status ok
Most containers will be restarted about every 50 minutes.
I have no idea about how to bring it back online. Is there any kind of checklist recomanded to prevent further damage, what should I do with this, how can I fix it.
check mon log
mon.ceph01 error log is like this:
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: -1> 2023-02-13T07:38:59.399+0000 7f49cfb95880 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/kv/RocksDBStore.cc: In function 'virtual int RocksDBStore::get(const string&, const string&, ceph::bufferlist*)' thread 7f49cfb95880 time 2023-02-13T07:38:59.399608+0000
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/kv/RocksDBStore.cc: 1899: ceph_abort_msg("block checksum mismatch: stored = 3293112312, computed = 2419631843 in /var/lib/ceph/mon/ceph-ceph01/store.db/214973.sst offset 18759843 size 3815")
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]:
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x7f49cf062a9f]
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 2: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3b3) [0x55960fc70453]
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 3: main()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 4: __libc_start_main()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 5: _start()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]:
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 0> 2023-02-13T07:38:59.401+0000 7f49cfb95880 -1 *** Caught signal (Aborted) **
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: in thread 7f49cfb95880 thread_name:ceph-mon
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]:
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f49cd00dcf0]
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 2: gsignal()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 3: abort()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x7f49cf062b5f]
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 5: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3b3) [0x55960fc70453]
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 6: main()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 7: __libc_start_main()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: 8: _start()
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Feb 13 15:38:59 ceph01.pinkcloud.localdomain ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01[2403]:
Feb 13 15:38:59 ceph01.pinkcloud.localdomain podman[2437]: 2023-02-13 15:38:59.680818197 +0800 CST m=+0.070475611 container died da69765578771a381ad9d84870d97a167b601f75a2f13fde9fb55f958d1811bc (image=quay.io/ceph/ceph#sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45, name=ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01)
Feb 13 15:38:59 ceph01.pinkcloud.localdomain podman[2437]: 2023-02-13 15:38:59.717834149 +0800 CST m=+0.107491608 container remove da69765578771a381ad9d84870d97a167b601f75a2f13fde9fb55f958d1811bc (image=quay.io/ceph/ceph#sha256:0560b16bec6e84345f29fb6693cd2430884e6efff16a95d5bdd0bb06d7661c45, name=ceph-bf74b922-5d17-11ed-977a-525400b60738-mon-ceph01, io.openshift.tags=base centos centos-stream, release=943, maintainer=Guillaume Abrioux <gabrioux#redhat.com>, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-943, vcs-type=git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, ceph=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, build-date=2022-09-06T17:25:04.845275, GIT_COMMIT=f77ca5de7910f1e3de260a1218c757954afd8327, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, name=centos-stream, version=8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.5, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image
tried restart ceph.target, but it doesn't help anything.
how to repair this "block checksum mismatch" thing?

Answer 1

If all servers went down at the same time the data should be fine except for some pending writes. But the overall integrity should be intact. You need to start the MONs first so they form a quorum and other services and clients can connect. If the MONs fail to start check the MON logs with cephadm logs --name mon.<YOUR_MON>, and the same for other services. Basically, try to start in the reverse order of a regular shutdown procedure, one example from the SUSE docs.