mongod fails to start with error message "missing UUID" - mongodb

Question

Mongod has been running as v4 for several months. It is used to record clinical data for patient with neurological diseases, and is essential to the drug trials.
This morning mongo crashed, and upon restarting, mongod refuses to start, on a collection that has a missing UUID. The message is cryptic, as this is not a 3.6 DB. Is there a way to proceed? The sensorTag collection is not really essential, and erasing it would be OK. Is there a way to get mongod to use the DB at all?
Here is the startup message:
2020-05-18T23:23:58.587+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] MongoDB starting : pid=1526 port=23417 dbpath=./db 64-bit host=retro-4-10-19
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] db version v4.0.18
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] git version: 6883bdfb8b8cff32176b1fd176df04da9165fd67
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] modules: none
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] build environment:
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] distmod: ubuntu1804
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] distarch: x86_64
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-05-18T23:23:58.590+0000 I CONTROL [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 23417 }, storage: { dbPath: "./db" } }
2020-05-18T23:23:58.591+0000 I STORAGE [initandlisten] Detected data files in ./db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-05-18T23:23:58.591+0000 I STORAGE [initandlisten]
2020-05-18T23:23:58.591+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-05-18T23:23:58.591+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-05-18T23:23:58.591+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=484M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-05-18T23:23:59.519+0000 I STORAGE [initandlisten] WiredTiger message [1589844239:519947][1526:0x7fa98a9fca40], txn-recover: Main recovery loop: starting at 46/768 to 47/256
2020-05-18T23:23:59.635+0000 I STORAGE [initandlisten] WiredTiger message [1589844239:635147][1526:0x7fa98a9fca40], txn-recover: Recovering log 46 through 47
2020-05-18T23:23:59.703+0000 I STORAGE [initandlisten] WiredTiger message [1589844239:703356][1526:0x7fa98a9fca40], txn-recover: Recovering log 47 through 47
2020-05-18T23:23:59.767+0000 I STORAGE [initandlisten] WiredTiger message [1589844239:767605][1526:0x7fa98a9fca40], txn-recover: Set global recovery timestamp: 0
2020-05-18T23:23:59.794+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2020-05-18T23:23:59.807+0000 I CONTROL [initandlisten]
2020-05-18T23:23:59.807+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-05-18T23:23:59.807+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2020-05-18T23:23:59.807+0000 I CONTROL [initandlisten]
2020-05-18T23:23:59.823+0000 F CONTROL [initandlisten] ** IMPORTANT: Collection RetrotopeClinic.sensortags is missing an UUID. We expect all collections to have UUIDs starting in FCV 3.6. Please make sure the FCV is version 3.6 before attempting an upgrade to 4.0; see http://dochub.mongodb.org/core/4.0-upgrade-fcv for more details. If the FCV is already 3.6, please try --repair with a 3.6 binary or initial sync to fix the data files.
2020-05-18T23:23:59.823+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2020-05-18T23:23:59.823+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-23417.sock
2020-05-18T23:23:59.823+0000 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2020-05-18T23:23:59.823+0000 I STORAGE [initandlisten] Shutting down session sweeper thread
2020-05-18T23:23:59.823+0000 I STORAGE [initandlisten] Finished shutting down session sweeper thread
2020-05-18T23:23:59.901+0000 I STORAGE [initandlisten] Downgrading WiredTiger datafiles.
2020-05-18T23:24:00.088+0000 I STORAGE [initandlisten] WiredTiger message [1589844240:88483][1526:0x7fa98a9fca40], txn-recover: Main recovery loop: starting at 47/2176 to 48/256
2020-05-18T23:24:00.190+0000 I STORAGE [initandlisten] WiredTiger message [1589844240:190325][1526:0x7fa98a9fca40], txn-recover: Recovering log 47 through 48
2020-05-18T23:24:00.250+0000 I STORAGE [initandlisten] WiredTiger message [1589844240:250635][1526:0x7fa98a9fca40], txn-recover: Recovering log 48 through 48
2020-05-18T23:24:00.302+0000 I STORAGE [initandlisten] WiredTiger message [1589844240:302709][1526:0x7fa98a9fca40], txn-recover: Set global recovery timestamp: 0
2020-05-18T23:24:00.477+0000 I STORAGE [initandlisten] shutdown: removing fs lock...
2020-05-18T23:24:00.477+0000 I CONTROL [initandlisten] now exiting
2020-05-18T23:24:00.477+0000 I CONTROL [initandlisten] shutting down with code:62
I did also run with the --repair flag: it said it repaied the sensorTag collection, but no, it didn't, and still failed:
020-05-19T00:09:27.800+0000 I STORAGE [initandlisten] Repairing collection StagServExpress.sensortags
2020-05-19T00:09:27.801+0000 I STORAGE [initandlisten] Verify succeeded on uri table:collection-22-4256855214379632921. Not salvaging.
2020-05-19T00:09:27.805+0000 I INDEX [initandlisten] build index on: StagServExpress.sensortags properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "StagServExpress.sensortags" }
2020-05-19T00:09:27.806+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 166 megabytes of RAM
2020-05-19T00:09:27.810+0000 I INDEX [initandlisten] build index on: StagServExpress.sensortags properties: { v: 2, unique: true, key: { UUID: 1 }, name: "UUID_1", ns: "StagServExpress.sensortags", background: true }
2020-05-19T00:09:27.810+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 166 megabytes of RAM
2020-05-19T00:09:27.813+0000 I INDEX [initandlisten] build index on: StagServExpress.sensortags properties: { v: 2, key: { nickname: 1 }, name: "nickname_1", ns: "StagServExpress.sensortags", background: true }
2020-05-19T00:09:27.813+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 166 megabytes of RAM

Answer 1

MongoDB 4.0.18 was released a month ago (April 13). You said:
Mongod has been running as v4 for several months.
Which version were you running for "several months" and how long have you been running 4.0.18 for?
Did you upgrade MongoDB at the same time as you experienced the crash?
Do you have backups?
Assuming you have a backup, reinstate the version of MongoDB that was in use prior to the crash and run it on the data directory as close to the crash as your backups allow.
The error message you are seeing was added in 4.0.13, so if you run 4.0.12 or earlier you shouldn't get this error (but I don't know if your data is corrupted and what other issues might exist).
Functionality does get backported to older versions, so it's possible there are older 4.0 servers that don't have the UUID that the current 4.0 is expecting because the changes were made in 4.0 (or 4.2) and backported to 3.6 but after 4.0.0 came out.