I am new to MongoDB. I want to copy some old data from one database to another new database.
I wrote a script that works, but I want to same change in my old data.
Present my data format is:
user {
_id: 1,
username: "jon",
email: "jon#gmail.com"
}
But I want this format:
user {
_id: "621721aed6d2481c999429e5",
username: "jon",
email: "jon#gmail.com",
old_id: 1
}
My script is:
var bulk = db.getSiblingDB("blog-new")["users"].initializeUnorderedBulkOp();
db.getCollection("users").find().forEach(function (d) {
bulk.insert(d);
});
bulk.execute();
How can I solve this issue?
Update
solve this issue using this script-
var bulk = db.getSiblingDB("user-new")["users"].initializeUnorderedBulkOp();
db.getCollection("users").find().forEach(function (d) {
d.old_id = NumberInt(d._id)
d._id = ObjectId()
print("user", d);
bulk.insert(d);
});
bulk.execute();
Related
Iam trying to update my data using mongodb query in my nestjs project. But the data is not updating as required.
let school = await this.schoolModel.updateOne({ id: item.school, "exams.id": exam.id }, {
$set: {
"exams.$.terms": item.terms, "exams.$.exam": exam.id,
"exams.$.subject": item.subject
}
})
In my MongoDB/Node backend environment I am using Mongoose pre and post hook middleware to check what's changed on the document, in order to create some system notes as a result.
One problem I'm running into is that when I try and lookup the record for the document in question I get a "Customer.findOne()" is not a function error. This is ONLY a problem when I'm looking up a record from the same collection from which the model just launched this pre and post hook triggers file. In other words, if my "Customer" model kicks off functions in a pre hook function in an external file, then I get an error if I then try and lookup a Customer with a standard findOne():
My customer model looks something like this:
module.exports = mongoose.model(
"Customer",
mongoose
.Schema(
{
__v: {
type: Number,
select: false
},
deleted: {
type: Boolean,
default: false
},
// Other props
searchResults: [
{
matchKey: String,
matchValue: String
}
]
},
{
timestamps: true
}
)
.pre("save", function(next) {
const doc = this;
trigger.preSave(doc);
next();
})
.post("save", function(doc) {
trigger.postSave(doc);
})
.post("update", function(doc) {
trigger.postSave(doc);
})
.post("findOneAndUpdate", function(doc) {
trigger.postSave(doc);
})
);
... the problematic findOne() function in the triggers file being called from the model looks like this:
const Customer = require("../../models/customer");
exports.preSave = async function(doc) {
this.preSaveDoc = await Customer.findOne({
_id: doc._id
}).exec();
};
To clarify, this is NOT a problem if I'm using a findOne() to lookup a record from a different collection in this same triggers file. Then it works fine. See below when finding a Contact -- no problem here:
const Contact = require("../../models/contact");
exports.preSave = async function(doc) {
this.preSaveDoc = await Contact.findOne({
_id: doc._id
}).exec();
};
The workaround I've found is to use Mongo instead of Mongoose, like so:
exports.preSave = async function(doc) {
let MongoClient = await require("../../config/database")();
let db = MongoClient.connection.db;
db.collection("customers")
.findOne({ _id: doc._id })
.then(doc => {
this.preSaveDoc = doc;
});
}
... but I'd prefer to use Mongoose syntax here. How can I use a findOne() in a pre-hook function being called from the same model/collection as the lookup type?
I have ran similar issue few days ago.
Effectively it is a circular dependency problem. When you call .findOne() on your customer model it doesn't exist as it is not exported yet.
You should probably try something like that :
const customerSchema = mongoose.Schema(...);
customerSchema.pre("save", async function(next) {
const customer = await Customer.findOne({
_id: this._id
}).exec();
trigger.setPreSaveDoc(customer);
next();
})
const Customer = mongoose.model("Customer", customerSchema)
module.export Customer;
Here customer will be defined because it is not called (the pre hook) before its creation.
As an easier way (I am not sure about it) but you could try to move the Contact import in your Trigger file under the save function export. That way I think the decencies may works.
Did it helps ?
I have a mongodb with my users in it. Users are like that :
_id: ObjectId,
created_at: Date,
username: String
I don't know why but all my users created_at fields have been reset.
I have a save but i can't just use my save and replace all my users by the save because i always have new users. I just need to replace all existing users created_at field by their own created_at field in the save.
I can use mongoose and node.js to do that (i don't know how but i can find) But i would prefer to do that using only mongodb functions. My saves are on .bson format.
You could create a local database, put your .bson in it. Then do a request to your local db to get all users and save theme in a .json file. Then update your user with wrong date with your saved datas.
Puting your bson file in your database
mongorestore -d yourdatabase -c yourcollection "users.bson"
Set all your good users in a json file
db.model('User').find({}, { created_at: 1 }).exec(function(error, _users) {
jsonUsers = _users;
jsonUsers = JSON.stringify(jsonUsers);
fs.writeFile('myjsonfile.json', jsonUsers, 'utf8', (err) => {
});
});
Update your wrong users to be good again
const jsonUsers = require('../myjsonfile.json');
db.model('User').find({}, { created_at: 1 }).exec(function(error, _users) {
if (error) return ;
_users.forEach((wrongUser) => {
let goodUser = jsonUsers.find((gU) => { return gU._id == wrongUser._id });
if (goodUser) {
wrongUser.created_at = goodUser.created_at;
wrongUser.save(cb);
} else {
//this is on of your new user that where not in your .bson file
}
});
});
So from time to time I get new exports of a cities database of POIs and info about them and I want to have all that data in my MongoDB with a Loopback-API on it. Therefore I reduce the data to my desired structure and try to import it.
For the first time I receive such an export, I can simply insert the data with insertMany().
When I get a new export, it means that it includes updated POIs which I actually want my existing POIs to be replaced with that new data. So I thought I'd use updateMany() but I could'nt figure out how I'd do that in my case.
Here's what I have so far:
const fs = require('fs');
const mongoose = require('mongoose');
const data = JSON.parse(fs.readFileSync('data.json', 'utf8'));
// Connect to database
mongoose.connect('mongodb://localhost/test', {
useMongoClient: true
}, (err) => {
if (err) console.log('Error', err);
});
// Define schema
let poiSchema = new mongoose.Schema({
_id: Number,
name: String,
geo: String,
street: String,
housenumber: String,
phone: String,
website: String,
email: String,
category: String
});
// Set model
let poi = mongoose.model('poi', poiSchema);
// Generate specified data from given export
let reducedData = data['ogr:FeatureCollection']['gml:featureMember'].reduce((endData, iteratedItem) => {
endData = endData.length > 0 ? endData : [];
endData.push({
_id: iteratedItem['service']['fieldX'],
name: iteratedItem['service']['fieldX'],
geo: iteratedItem['service']['fieldX']['fieldY']['fieldZ'],
street: iteratedItem['service']['fieldX'],
housenumber: iteratedItem['service']['fieldX'],
phone: iteratedItem['service']['fieldX'],
website: iteratedItem['service']['fieldX'],
email: iteratedItem['service']['fieldX'],
category: iteratedItem['service']['fieldX']
});
return endData;
}, []);
//
// HERE: ?!?!? Insert/update reduced data in MongoDB collection ?!?!?
//
mongoose.disconnect();
So I just want to update everything that has changed.
Of course if I leave it to insertMany() it fails due to dup key.
For the second time, use mongo's update command with upsert set to true.
db.collection.update(query, update, options)
In the query pass the _id ,in update pass the object and in option set upsert to true. This will update the document if it exists creates a new document if that doesn't exist.
In the following link from mongoDB documentation:
https://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
there is an explanation on how to build a "safe" insert with auto-incremental _id.
My questions are:
Where should i define the function at?
How can i call it later on?
I couldn't quit understand if the function is stored as a JS function is a JS file? or is it stored in the DB? or something else.
Thanks.
Here is the actual function as it is brought in the link above:
function insertDocument(doc, targetCollection) {
while (1) {
var cursor = targetCollection.find( {}, { _id: 1 } ).sort( { _id: -1 } ).limit(1);
var seq = cursor.hasNext() ? cursor.next()._id + 1 : 1;
doc._id = seq;
var results = targetCollection.insert(doc);
if( results.hasWriteError() ) {
if( results.writeError.code == 11000 /* dup key */ )
continue;
else
print( "unexpected error inserting data: " + tojson( results ) );
}
break;
}
}
Implementation approach
Here's an example of how to implement an auto-increment field with mongoose:
var CounterSchema = Schema({
_id: {type: String, required: true},
seq: { type: Number, default: 0 }
});
var counter = mongoose.model('counter', CounterSchema);
var entitySchema = mongoose.Schema({
testvalue: {type: String}
});
entitySchema.pre('save', function(next) {
var doc = this;
counter.findByIdAndUpdate({_id: 'entityId'}, {$inc: { seq: 1} }, function(error, counter) {
if(error)
return next(error);
doc.testvalue = counter.seq;
next();
});
});
Source: https://stackoverflow.com/a/30164636/236660
Please note that {_id: 'entityId'} should be set differently per collection you're using this code with. So if you're generating an ID for entitySchema, you'll have {_id: 'entityId'}. For userSchema you would use {_id: 'userId'}, etc. You need to do this so that every collection has its own auto-incremented sequence.
The code above is thread safe. The findByIdAndUpdate operation is atomic; its consistency and concurrency are handled by MongoDB.
Existing module
Also, there is a mongoose module created specifically for handling auto-incremented IDs. You may actually be better off using it:
https://www.npmjs.com/package/mongoose-auto-increment