Unable to start Ceph RGW service - ceph

Question

Context:
I'm trying to setup a new ceph-17.2.5 cluster
Initial setup consists of 1 mgr, 1 mon and 1 osd all installed on same box using cephadm.
So far I am able to access http://192.168.3.50:8080/api/auth and get token in the response.
But http://192.168.3.50:8080/api/rgw/bucket returns:
`
{
"status": "500 Internal Server Error",
"detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.",
"request_id": "f08247f9-d9ea-4f88-99b7-d07d5141d5ea"
}
`
This seems because the dashboard shows rgw service is not running, I have tried to setup rgw service using below commands:
sudo cephadm shell --mount radosgw.yml:/var/lib/ceph/radosgw/radosgw.yml
ceph orch apply -i /var/lib/ceph/radosgw/radosgw.yml
radosgw.yml file contents:
service_type: rgw
service_id: default
placement:
hosts:
- <machine-hostname>
count_per_host: 1
spec:
rgw_realm: default
rgw_zone: default
rgw_frontend_port: 8123
networks:
- 192.168.3.0/24
After running above the cluster in ceph dashboard does show one rgw service:
But the object gateway daemon still gives error as below:
The RGW API still returns same error.
Looking at the rgw logs I see rgw keeps trying restart on a loop and eventually stops:
Nov 17 05:31:16 <hostname> systemd[1]: Started Ceph rgw.default.<hostname>.rerude for 76ec21dc-4f10-11ed-8fda-0050568dc50c. Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0 0 deferred set uid:gid to 167:167 (ceph:ceph) Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process radosgw, pid 7 Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0 0 framework: beast Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0 0 framework conf key: endpoint, val: 192.168.3.59:8123 Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.012+0000 7fc0de7ed5c0 1 radosgw_Main not setting numa affinity Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.015+0000 7fc0de7ed5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 Nov 17 05:31:17 <hostname> bash[2537663]: debug 2022-11-17T05:31:17.015+0000 7fc0de7ed5c0 1 D3N datacache enabled: 0 Nov 17 05:36:17 <hostname> bash[2537663]: debug 2022-11-17T05:36:17.013+0000 7fc0dd36f700 -1 Initialization timeout, failed to initialize Nov 17 05:36:17 <hostname> systemd[1]: ceph-76ec21dc-4f10-11ed-8fda-0050568dc50c#rgw.default.<hostname>.rerude.service: Main process exited, code=exited, status=1/FAILURE Nov 17 05:36:17 <hostname> systemd[1]: ceph-76ec21dc-4f10-11ed-8fda-0050568dc50c#rgw.default.<hostname>.rerude.service: Failed with result 'exit-code'. Nov 17 05:36:27 <hostname> systemd[1]: ceph-76ec21dc-4f10-11ed-8fda-0050568dc50c#rgw.default.<hostname>.rerude.service: Service RestartSec=10s expired, scheduling restart. Nov 17 05:36:27 <hostname> systemd[1]: ceph-76ec21dc-4f10-11ed-8fda-0050568dc50c#rgw.default.<hostname>.rerude.service: Scheduled restart job, restart counter is at 4. Nov 17 05:36:27 <hostname> systemd[1]: Stopped Ceph rgw.default.<hostname>.rerude for 76ec21dc-4f10-11ed-8fda-0050568dc50c. Nov 17 05:36:27 <hostname> systemd[1]: Started Ceph rgw.default.<hostname>.rerude for 76ec21dc-4f10-11ed-8fda-0050568dc50c. Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0 0 deferred set uid:gid to 167:167 (ceph:ceph) Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process radosgw, pid 6 Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0 0 framework: beast Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0 0 framework conf key: endpoint, val: 192.168.3.50:8123 Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.465+0000 7f2589bf95c0 1 radosgw_Main not setting numa affinity Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.468+0000 7f2589bf95c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 Nov 17 05:36:27 <hostname> bash[2538424]: debug 2022-11-17T05:36:27.468+0000 7f2589bf95c0 1 D3N datacache enabled: 0 Nov 17 05:41:27 <hostname> bash[2538424]: debug 2022-11-17T05:41:27.466+0000 7f258877b700 -1 Initialization timeout, failed to initialize
> sudo ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 8 GiB 8.0 GiB 9.7 MiB 9.7 MiB 0.12
TOTAL 8 GiB 8.0 GiB 9.7 MiB 9.7 MiB 0.12
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.rgw.root 1 32 0 B 0 0 B 0 2.5 GiB
.mgr 2 1 449 KiB 2 452 KiB 0 7.6 GiB
Not sure what should I look for to get the rgw apis to work in a minimal setup.