I'm trying to reconcile Ceph OSD encryption workflow
OSD is created, both lockbox and dmcrypt keys are created, and sent along with JSON to the monitors, indicating an encrypted OSD.
All complementary devices (like journal, db, or wal) get created and encrypted with the same OSD key. Key is stored in the LVM metadata of the OSD
Activation continues by ensuring devices are mounted, retrieving the dmcrypt secret key from the monitors and decrypting before the OSD gets started.
The (2) step is confusing as it talks about the "OSD key", which is stored in LVM metadata. If the dmcrypt key is stored in the monitor, what is the OSD key? "The same OSD key" means the same as dmcrypt key?
Related
I have mongodb (version 4.2) replicaset with 3 nodes - primary, secondary, arbiter,
primary occupies close to 250 GB disk space, oplog size is 15 GB
secondary was down for few hours, tried recovering it by restarting, it went into recovering forever.
tried initial sync by deleting files on data path, took 15 hours, data path size went to 140GB and failed
tried to copy files from primary and seed it to recover secondary node
followed https://www.mongodb.com/docs/v4.2/tutorial/resync-replica-set-member/
This did not work - (again stale)
in the latest doc (5.0) they mention to use a new member ID, does it apply for 4.2 as well?
changing the member ID throws error as IP and port is same for node I am trying to recover
This method was also unsuccessful, planning to recover the node using different data path and port as primary might consider it as a new node, then once the secondary is up, will change the port to which I want and restart, will it work?
please provide any other suggestions to recover a replica node with large data like 250 GB
shut down primary
Copying the data files from primary node, placing it in new db path (other than the recovering nodes db path)
changing log path
starting mongo service with different port (other than the one used by recovering node)
start primary
adding it to replicaset using rs.add("IP:new port") on primary
worked, could see the secondary node coming up successfully
The Cloud SQL encryption docs (https://cloud.google.com/sql/docs/sqlserver/cmek#when_does_interact_with_cmek_keys) state:
Read replicas from a CMEK-enabled instance inherit CMEK encryption with the same Cloud KMS key as the primary instance.
At the same time:
Note: The Cloud KMS key ring location must match the region where you want to create a Cloud SQL instance. A multi-region or global region key will not work. A request for creating a Cloud SQL instance fails if the regions don't match.
From these two pieces of information one could conclude that cross region replicas are not possible
to be used alongside CMEK encryption.
However, we've labbed this through by:
creating a KMS keyring + key in europe-west3 and a Cloud SQL primary instance in europe-west3 using that key
creating a KMS keyring + key in europe-west2 and a Cloud SQL replica in europe-west2 using the key from europe-west2 (replica for the abovementioned primary)
Can we rely on what we've labbed in practice? Are the docs inaccurrate?
Answer can be found on a different doc page:
When you create a read replica of a Cloud SQL instance in the same region, it inherits the same customer-managed encryption key as the parent instance. If you create a read replica in a different region, you are given a new list of customer-managed encryption keys to select from. Each region uses its own set of keys.
Yesterday my teammates found a way to disable cephx authentication cluster wide (2 server cluster) in order to bypass issues that were preventing us from joining a 3rd server. Unfortunately they were uncertain which of the steps taken let to the successful addition. I request assistance getting my ceph operational again. Yesterday we left off after editing /etc/ceph/ceph.conf, turning authentication back on here, then copying the file to /var/lib/ceph///config and ensuring permissions were set to 644.
This got one command to work that had previously not been -- my ceph osd df correctly shows all 24 OSDs again, but I cannot run a ceph osd status nor a ceph orch status.
I am using Ceph, uploading many files through radosgw. After, I want to delete the files. I am trying to do that in Python, like this:
bucket = conn.get_bucket(BUCKET)
for key in bucket.list():
bucket.delete_key(key)
Afterwards, I use bucket.list() to list files in the bucket, and this says that the bucket is now empty, as I intended.
However, when I run ceph df on the mon, it shows that the OSDs still have high utilization (e.g. %RAW USED 90.91). If I continue writing (thinking that the status data just hasn't caught up with the state yet), Ceph essentially locks up (100% utilization).
What's going on?
Note: I do have these standing out in ceph status:
health HEALTH_WARN
3 near full osd(s)
too many PGs per OSD (2168 > max 300)
pool default.rgw.buckets.data has many more objects per pg than average (too few pgs?)
From what I gather online, this wouldn't cause my particular issue. But I'm new to Ceph and could be wrong.
I have one mon and 3 OSDs. This is just for testing.
You can check if the object is really deleted by rados -p $pool list,
I knew for cephfs, when you delete a file, it will return ok when mds mark
it as deleted in local memory and then do real delete by sending delete messages to related osd.
Maybe radosgw use the same design to speed up delete
I am getting different things from different sources. At first I thought that I could just have one configuration file (on the monitor), with sections for each node (include osd nodes). But when on the new OSD, ceph osd create fails, saying there is no configuration file.
So, how does the configuration structure of ceph work? Further, is the FSID (a UUID) in each configuration file the same?
Yes, every machine planned to deploy osd/mon/mds should have a /etc/ceph/ceph.conf file.
When creating osd instance, it needs to communicate with mon from configure file. The fsid should be same, as osd/mds/mon will compare the fsid when handling internal messages; if they are not the same, the message will be dropped.