1/3 OSD down in Ceph Cluster after 95% of the storage consume - ceph

Question

I'm new to Ceph technology so I may not know obvious stuff. I started deploying ceph cluster using cephadm and I did. In my first attempt I gave each node 3 GB RAM (After some time I figured out that it needs more). My cluster hanged when one of the node's ram and swap filled up to 100%. Now I give each node 8GB ram and 10GB SSD swap and It's fixed:
Node01:
4x CPU , 8GB RAM, 60GB SSD
Node02:
4x CPU , 6GB RAM , 60GB SSD
Node04:
4x CPU , 8GB RAM , 60GB HDD
I start using it by creating a CephFS (It creates 2 pools one for data and one for metadata (3x replica rule)). I mount this FS on an Ubuntu 20.04 using ceph-common:
>> vim /etc/fstab
...
ceph-n01,ceph-n02:/ /ceph ceph _netdev,name=amin,secretfile=/home/amin/.ceph 0 0
It works fine. I use this fs by running a service that render a map and save the tiles in the filesystem (my CephFS pool). It works for about 1 day and a half and generates ~56.65GB file). On the second day I saw that 1 OSD (OSD with HDD) and only two OSDs running.
I checked RAM and CPU status for the 3 nodes. In 2 nodes 50% of the RAM was used and in one node (node 01) 85% of the RAM was used with ~4GB swap. I tried to fix the issue by restarting the OSD. the OSD which was down kept crashing when I restarted them. (the OSDs which was running before, started successfully after restart.
I looked at OSD logs:
debug -11> 2022-01-06T12:43:01.620+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981624364, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 3219, "file_size": 6014, "table_properties": {"data_size": 4653, "index_size": 64, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 2663, "raw_average_key_size": 17, "raw_value_size": 2408, "raw_average_value_size": 16, "num_data_blocks": 2, "num_entries": 148, "num_deletions": 0, "num_merge_operands": 147, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": ".T:int64_array.b:bitwise_xor", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -10> 2022-01-06T12:43:01.652+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981656222, "cf_name": "p-0", "job": 1, "event": "table_file_creation", "file_number": 3220, "file_size": 5104328, "table_properties": {"data_size": 4982595, "index_size": 70910, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 49989, "raw_key_size": 973446, "raw_average_key_size": 48, "raw_value_size": 4492366, "raw_average_value_size": 224, "num_data_blocks": 1298, "num_entries": 19980, "num_deletions": 10845, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "p-0", "column_family_id": 4, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -9> 2022-01-06T12:43:01.688+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981692873, "cf_name": "p-2", "job": 1, "event": "table_file_creation", "file_number": 3221, "file_size": 5840600, "table_properties": {"data_size": 5701923, "index_size": 81198, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 56645, "raw_key_size": 1103994, "raw_average_key_size": 48, "raw_value_size": 5146222, "raw_average_value_size": 227, "num_data_blocks": 1485, "num_entries": 22623, "num_deletions": 12166, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "p-2", "column_family_id": 6, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -8> 2022-01-06T12:43:01.688+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981694121, "cf_name": "O-0", "job": 1, "event": "table_file_creation", "file_number": 3222, "file_size": 73885, "table_properties": {"data_size": 72021, "index_size": 588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 9444, "raw_average_key_size": 60, "raw_value_size": 63028, "raw_average_value_size": 406, "num_data_blocks": 18, "num_entries": 155, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "O-0", "column_family_id": 7, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -7> 2022-01-06T12:43:01.688+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981695243, "cf_name": "O-1", "job": 1, "event": "table_file_creation", "file_number": 3223, "file_size": 71023, "table_properties": {"data_size": 69158, "index_size": 589, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 9028, "raw_average_key_size": 61, "raw_value_size": 60508, "raw_average_value_size": 408, "num_data_blocks": 18, "num_entries": 148, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "O-1", "column_family_id": 8, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -6> 2022-01-06T12:43:01.692+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981696397, "cf_name": "O-2", "job": 1, "event": "table_file_creation", "file_number": 3224, "file_size": 75263, "table_properties": {"data_size": 73370, "index_size": 617, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 9679, "raw_average_key_size": 60, "raw_value_size": 64238, "raw_average_value_size": 404, "num_data_blocks": 19, "num_entries": 159, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "O-2", "column_family_id": 9, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -5> 2022-01-06T12:43:01.696+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981700113, "cf_name": "L", "job": 1, "event": "table_file_creation", "file_number": 3225, "file_size": 338198, "table_properties": {"data_size": 335953, "index_size": 1100, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 325, "raw_key_size": 1712, "raw_average_key_size": 16, "raw_value_size": 333803, "raw_average_value_size": 3119, "num_data_blocks": 39, "num_entries": 107, "num_deletions": 68, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "L", "column_family_id": 10, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -4> 2022-01-06T12:43:01.784+0000 7feaa5390080 1 bluefs _allocate unable to allocate 0x80000 on bdev 1, allocator name block, allocator type bitmap, capacity 0xeffc00000, block size 0x1000, free 0x10c7c2000, fragmentation 1, allocated 0x0
debug -3> 2022-01-06T12:43:01.784+0000 7feaa5390080 -1 bluefs _allocate allocation failed, needed 0x7b2e8
debug -2> 2022-01-06T12:43:01.784+0000 7feaa5390080 -1 bluefs _flush_range allocated: 0xc90000 offset: 0xc8a944 length: 0x809a4
debug -1> 2022-01-06T12:43:01.792+0000 7feaa5390080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7feaa5390080 time 2022-01-06T12:43:01.789216+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc")
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x558699f6ac8c]
2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x55869a661901]
3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55869a661be0]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x32) [0x55869a672cf2]
5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x55869a68b32b]
6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55869ab1dacf]
7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x55869ac2f81a]
8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x55869ac30c70]
9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x55869ad4c416]
10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x55869ad4cd5c]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x55869ad4d47c]
12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x55869ad4d50d]
13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x55869ad50978]
14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x55869acfb3d5]
15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x55869ab60415]
16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1c2e) [0x55869ab62b4e]
17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x55869ab63ea8]
18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x59d) [0x55869ab5dbcd]
19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x15) [0x55869ab5ef65]
20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10c1) [0x55869aad6ec1]
21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x55869a55c4d8]
22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x55869a5c6657]
23: (BlueStore::_mount()+0x204) [0x55869a5c9514]
24: (OSD::init()+0x380) [0x55869a09ea10]
25: main()
26: __libc_start_main()
27: _start()
debug 0> 2022-01-06T12:43:01.796+0000 7feaa5390080 -1 *** Caught signal (Aborted) **
in thread 7feaa5390080 thread_name:ceph-osd
the above is the log in the down OSD.
I start reading it and find a useful log to search in google:
bluefs _allocate unable to allocate 0x80000 on bdev 1, allocator name block, allocator type bitmap, capacity 0xeffc00000, block size 0x1000, free 0x10c7c2000, fragmentation 1, allocated 0x0
I found a bug which was related to version 16.2.1 (I use 16.2.6):
https://tracker.ceph.com/issues/50656
I wanted to get a dump from my OSD (I dont understand completly what they say):
ceph daemon osd.1 bluestore allocator dump block
Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n"
I deploy cluster using cephadm which uses containers so I can not access socket in this fashion I think. this command leads me to use ceph-bluestore-tool to see state of my physical disk (see capacity, run fsck or repair) but running ceph-bluestore-tool needs to specify the --path of the osd which I cant run it from host (my containers keep crashing so I can't run this command inside the container). I tried to run a command in osd container using cephadm but can't find anyway to do this.
If you need the full log here tell me (I could not send it due to char limit) but it was the same crash log
I dont really understand whats going on.
I tried to use ceph-volume to mount the block device to the host to use ceph-bluestore-tool on it to use fsck or repair on it. (it needs a --path argument to point to the osd files) (I dont even know if it is the correct to use ceph-volume this way or it's built for this - as I told I am new to Ceph)
I tried to use cephadm to run ceph-bluestore-tool commands in crashed OSDs but I couldn't.
(the socket error I mentiond above)
my SSD OSDs was filled up 94% so the other still have free space on them (as I guess).
the only lead that I could find on the internet was not working.
I'm really desperate to find the answer. I will be happy if you can help me. even tell me to read a document or learn something.
I WILL POST SOME INFORMATION ABOUT MY CLUSTER DOWN HERE:
[The Ceph Dashboard][1]
>> ceph -s
cluster:
id: 1ad06d18-3e72-11ec-8684-fd37cdad1703
health: HEALTH_WARN
mons ceph-n01,ceph-n02,ceph-n04 are low on available space
2 backfillfull osd(s)
Degraded data redundancy: 4282822/12848466 objects degraded (33.333%), 64 pgs degraded, 96 pgs undersized
3 pool(s) backfillfull
6 daemons have recently crashed
services:
mon: 3 daemons, quorum ceph-n01,ceph-n04,ceph-n02 (age 7h)
mgr: ceph-n02.xyrntr(active, since 4w), standbys: ceph-n04.srrvqt
mds: 1/1 daemons up, 1 standby
osd: 3 osds: 2 up (since 6h), 2 in (since 6h)
data:
volumes: 1/1 healthy
pools: 3 pools, 96 pgs
objects: 4.28M objects, 42 GiB
usage: 113 GiB used, 6.7 GiB / 120 GiB avail
pgs: 4282822/12848466 objects degraded (33.333%)
64 active+undersized+degraded
32 active+undersized
>> ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 4m ago 8w count:1
crash 3/3 6m ago 8w *
grafana ?:3000 1/1 4m ago 8w count:1
mds.cephfs 2/2 6m ago 7w label:mds
mgr 2/2 6m ago 8w label:mgr
mon 3/5 6m ago 3w count:5
node-exporter ?:9100 3/3 6m ago 8w *
osd 3/3 6m ago - <unmanaged>
prometheus ?:9095 1/1 4m ago 8w count:1
>> ceph orch host ls
HOST ADDR LABELS STATUS
ceph-n01 192.168.2.20 _admin mon mds
ceph-n02 192.168.2.21 mon mgr
ceph-n04 192.168.2.23 _admin mon mds mgr
>> ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.17578 root default
-3 0.05859 host ceph-n01
1 ssd 0.05859 osd.1 up 1.00000 1.00000
-5 0.05859 host ceph-n02
0 ssd 0.05859 osd.0 up 1.00000 1.00000
-10 0.05859 host ceph-n04
2 hdd 0.05859 osd.2 down 0 1.00000
>> ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 ssd 0.05859 1.00000 60 GiB 57 GiB 50 GiB 2.1 GiB 4.6 GiB 3.4 GiB 94.37 1.00 96 up
0 ssd 0.05859 1.00000 60 GiB 57 GiB 50 GiB 2.1 GiB 4.6 GiB 3.4 GiB 94.41 1.00 96 up
2 hdd 0.05859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
TOTAL 120 GiB 113 GiB 100 GiB 4.3 GiB 9.1 GiB 6.7 GiB 94.39
MIN/MAX VAR: 1.00/1.00 STDDEV: 0.02
node01: >> free -m
total used free shared buff/cache available
Mem: 7956 7121 175 0 659 573
Swap: 10238 3748 6490
node02: >> free -m
total used free shared buff/cache available
Mem: 7956 7121 175 0 659 573
Swap: 10238 3748 6490
node04: >> free -m
total used free shared buff/cache available
Mem: 7922 2260 3970 1 1690 5371
Swap: 10238 642 9596