Cannot push more than one documents to collection (11000) - mongodb

This is my Operation Model:
const mongoose = require('mongoose');
const operationSchema = mongoose.Schema({
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User'
},
extract: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Extract'
},
normalizations: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Normalize'
}],
ingestions: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Ingest'
}]
});
module.exports = mongoose.model('Operation', operationSchema);
This is how I am creating an Operation:
exports.create = async (req, res, nexr) => {
let op = new Operation
op.user = mongoose.Types.ObjectId(req.body.user);
op.extract = mongoose.Types.ObjectId(req.body.extract);
op.normalizations = [];
op.ingestions = [];
const operation = new Operation(op);
try {
await operation.save();
console.log(operation)
res.send(operation);
} catch (err) {
console.log(operation)
res.status(500).send(err);
}
}
NO UNIQUE LABELS exists in any of the User, Extract, Normalize, Ingest Models
When I try to push to Mongo, the first time everything is ok.
If I try to push more than one records I get this error from Postman:
{
"driver": true,
"name": "MongoError",
"index": 0,
"code": 11000,
"keyPattern": {
"extract.name": 1
},
"keyValue": {
"extract.name": null
} }

only a single document in the collection can be missing the unique field.There is already a document in this collection with a missing or null value. The index is not present in the collection in which you are trying to insert. So try this, drop that collection from the database and run the program again.
Any documents missing the source_references.key field will be considered as having a null value. If you want the index to only apply to documents with a source_references.key field you can use sparse:true index creation option.

Related

Documents inserted without schema not being found with schema

I have two new collections in MongoDB of data that I pulled from an old Firestore database that I'm moving to mongo. Since the total number between these two collections is roughly 20,000, I opted to paste the raw JSON into the insert document section in mongo, which worked like a charm and I didn't have to write a new insert route to do the same.
I then created a schema in Mongoose that matched the inserted documents, and tried to use the schema to pull back some data, and its always returning nothing.
An example of a ticket inserted via JSON:
{
"title": "How to add and manage users for your company in QuickBooks Online",
"priority": "1",
"type": "Video",
"course": "G205",
"transcriptId": "07dom27Zz98jakvB1oh5",
"status": "In Review",
"tags": "",
"url": "",
"revisionNumber": 0,
"directoryId": 19,
"checkedOut": false
},
And my schema I made to match. The collection name in mongo is also called oldTickets, the plural of my schema name here:
const mongoose = require('mongoose');
var Schema = mongoose.Schema
const schema = new Schema({
course: { type: String },
title: { type: String },
priority: { type: String },
type: { type: String },
course: { type: String },
transcriptId: { type: String },
status: { type: String },
tags: { type: String },
url: { type: String },
revisionNumber: { type: Number },
directoryId: { type: Number },
checkedOut: { type: Boolean },
});
module.exports = mongoose.model('oldTicket', schema);
And finally my model import and fetch call:
const OldTicket = require('./models/model_old_ticket');
/***************************************************************************
* Get Old Tickets - Returns all old tickets, 10 at a time
****************************************************************************/
app.get('/getOldTickets/:offset', (req, res) => {
checkConnection();
OldTicket.find().skip(parseInt(req.params.offset)).limit(10).exec((err, data) => {
if (err){ res.status(500).send({err: err}); }
//If we got data, count the tickets & return the tickets & count
if (data) {
OldTicket.find().countDocuments().then(count => {
return res.status(200).send({
tickets: data,
count: count
})
})
}
});
});
Why isn't this finding anything? Both the count and the tickets are 0. I've run into this issue before when manually creating a collection without a schema, and in those instances I would simply delete the collection, write a route to create a document, and then things would work fine. But with the large data size of these two collections, I'd rather not do that since everything should be working as is.
Edit: Example of document in Mongo
And the name of the collection I'm currently viewing:
And I just now realized that for some reason there are now two collection names, oldTickets, which has data, and oldtickets, which is empty. I'm assuming my query is searching through the empty one? How can I get it to go to the one that actually has data?
can you attach the screenshot of your data with the collection? might be it's different.in mongoose, every collection name is complete with 's'. please verify your collection is created manually by you then it has to same as mongoose schema and also completed with 's'.
example:
const mongoose = require("mongoose");
const schema = new mongoose.Schema(
{
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
index: true
},
filmId: {
type: mongoose.Schema.Types.ObjectId,
index: true
},
filmType: {
type: String,
index: true
},
birthday: {
type: Date
},
age: {
type: Number
},
terms: {
type: Boolean
}
},
{
versionKey: false,
timestamps: true,
}
);
schema.index({ filmId: 1, user: 1 })
module.exports = mongoose.model("UserAgeVerification", schema);
see my database

GraphQL Mutation Updating Users Followers with Mongoose/MongodDB - $set is empty error

I have this mutation set up:
followUser: {
type: UserType,
args: {
_id: { type: GraphQLString },
firebaseUid: { type: GraphQLString },
following: { type: new GraphQLList(GraphQLString)},
},
resolve(parentValue, { firebaseUid, _id, following}) {
const update = {
$set: { "following": [firebaseUid] },
$push: { "following": { firebaseUid } }
}
return UserSchema.findOneAndUpdate(
{ _id },
update,
{new: true, upsert: true}
)
}
},
I'm trying to add new followers into my graphql user's collection. My user model:
const UserSchema = new Schema(
{
firebaseUid: String,
following: [{ type: Schema.Types.ObjectId, ref: 'User' }],
followers: [{ type: Schema.Types.ObjectId, ref: 'User' }],
},
{ timestamps: true }
);
module.exports = mongoose.model("User", UserSchema);
So at first, the user doesn't have any followers, so it won't have that field yet. When user adds someone to their friends list, thats when the field will appear in mongodb. Right now I'm getting this error:
"message": "'$set' is empty. You must specify a field like so: {$set: {<field>: ...}}",
I'm not sure if I'm doing the $set correctly.
The UserType
const UserType = new GraphQLObjectType({
name: "User",
fields: () => ({
_id: { type: GraphQLString },
firebaseUid: { type: GraphQLString },
following: { type: new GraphQLList(GraphQLString) },
followers: { type: new GraphQLList(GraphQLString) },
...
})
});
edit:
current mongodb data collection:
_id: ObjectId("5e5c24111c9d4400006d0001")
name: "Mr. Smith"
username: "mrsmith"
after running the update
_id: ObjectId("5e5c24111c9d4400006d0001")
name: "Mr. Smith"
username: "mrsmith"
following: ["fdsaduybfeaf323dfa"] // <-- this gets added
Currently mongooses validator is rejecting the update. To fix this you need the following:
You only need to $push since it will automatically create an array if the property does not exist
You should remove the extra { } around the firebaseUid in the $push because otherwise the following array will contain objects with a firebaseUid property instead of directly containing the Uid (or would if the schema validator allowed it)
Mongo ObjectIds can only be converted from strings when they are 12-byte hexadecimal, and firebaseUid is not, so the schema should be typed to String instead of ObjectId as the validator will reject the field for update otherwise.

Update query adding ObjectIDs to array twice

I am working on a table planner application where guests can be assigned to tables. The table model has the following Schema:
const mongoose = require('mongoose');
mongoose.Promise = global.Promise;
const tableSchema = new mongoose.Schema({
name: {
type: String,
required: 'Please provide the name of the table',
trim: true,
},
capacity: {
type: Number,
required: 'Please provide the capacity of the table',
},
guests: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Guest',
}],
});
module.exports = mongoose.model('Table', tableSchema);
Guests can be dragged and dropped in the App (using React DND) to "Table" React components. Upon being dropped on a table, an Axios POST request is made to a Node.js method to update the Database and add the guest's Object ID to an array within the Table model:
exports.updateTableGuests = async (req, res) => {
console.log(req.body.guestId);
await Table.findOneAndUpdate(
{ name: req.body.tablename },
{ $push: { guests: req.body.guestId } },
{ safe: true, upsert: true },
(err) => {
if (err) {
console.log(err);
} else {
// do stuff
}
},
);
res.send('back');
};
This is working as expected, except that with each dropped guest, the Table model's guests array is updated with the same guest Object ID twice? Does anyone know why this would be?
I have tried logging the req.body.guestID to ensure that it is a single value and also to check that this function is not being called twice. But neither of those tests brought unexpected results. I therefore suspect something is wrong with my findOneAndUpdate query?
Don't use $push operator here, you need to use $addToSet operator instead...
The $push operator can update the array with same value many times
where as The $addToSet operator adds a value to an array unless the
value is already present.
exports.updateTableGuests = async (req, res) => {
console.log(req.body.guestId);
await Table.findOneAndUpdate(
{ name: req.body.tablename },
{ $addToSet : { guests: req.body.guestId } },
{ safe: true, upsert: true },
(err) => {
if (err) {
console.log(err);
} else {
// do stuff
}
},
);
res.send('back');
};
I am not sure if addToSet is the best solution because the query being executed twice.
If you used a callback and a promise simultaneously, it would make the query executes twice.
So choosing one of them would make it works fine.
Like below:
async updateField({ fieldName, shop_id, item }) {
return Shop.findByIdAndUpdate(
shop_id,
{ $push: { menuItems: item } },
{ upsert: true, new: true }
);
}

Creating multiple documents in mongoose only if it does not currently exist

I was wondering if to create multiple documents in mongoose, but only if they do not exist currently? From the documentation, I've found the code below to create multiple documents, but just wondering how to ensure that it does not create a document if it currently exist?
In particular, if one document already exists, I would like the other documents that are not currently created to be created (rather than the entire create operation to fail).
From Documentation
var array = [{ type: 'jelly bean' }, { type: 'snickers' }];
Candy.create(array, function (err, jellybean, snickers) {
if (err) // ...
});
As noted in the documentation, the .create() method is a shortcut function for creating a new document for the given model and "saving" it to the collection. This actually works like the more formal .save() method but in shortcut form.
What you are describing though is more akin to the "upsert" behavior of the MongoDB .update() method. Which can also apply to its .findAndModify cousin or specifically in mongoose, the .findOneAndUpdate() method.
So with some sample code:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/nodetest');
var candySchema = new Schema({
type: String
});
var Candy = mongoose.model( "Candy", candySchema );
var array = [
{ type: 'jelly bean' },
{ type: 'snickers' },
{ type: 'mars' },
{ type: 'snickers' }
];
array.forEach(function(n) {
Candy.findOneAndUpdate( n, n, { upsert: true }, function(err,doc) {
console.log( doc );
});
});
You would see the following output:
{ _id: 535088e2e4beaab004e6cd97, type: 'jelly bean' }
{ _id: 535088e2e4beaab004e6cd98, type: 'snickers' }
{ _id: 535088e2e4beaab004e6cd99, type: 'mars' }
{ _id: 535088e2e4beaab004e6cd98, type: 'snickers' }
Noting that the second entry for 'snickers' actually refers to the object that was already created.
So that is a basic way to ensure that you are not actually creating the same data twice as long as you specify the "key" to match in the query condition.

Does Mongoose Actually Validate the Existence of An Object Id?

I like the validation that comes with Mongoose. We are trying to figure out whether we want to use it, and put up with the overhead. Does anyone know if providing a reference to the parent collection when creating a mongoose schema, (in the child schema, specify the object id of the parent object as a field,) does this then mean that every time you try to save the document it checks the parent collection for the existence of the refereneced object id?
I'm doing it with middleware, performing a search of the element on validation:
ExampleSchema = new mongoose.Schema({
parentId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Example'
}
});
ExampleModel = mongoose.model('Example', ExampleSchema);
ExampleSchema.path('parentId').validate(function (value, respond) {
ExampleModel.findOne({_id: value}, function (err, doc) {
if (err || !doc) {
respond(false);
} else {
respond(true);
}
});
}, 'Example non existent');
I'm using mongoose-id-validator. Works good
var mongoose = require('mongoose');
var idValidator = require('mongoose-id-validator');
var ReferencedModel = new mongoose.Schema({name: String});
var MySchema = new mongoose.Schema({
referencedObj : { type: mongoose.Schema.Types.ObjectId, ref: 'ReferencedModel'},
referencedObjArray: [{ type: mongoose.Schema.Types.ObjectId, ref: 'ReferencedModel' }]
});
MySchema.plugin(idValidator);
No, an ObjectId field that's defined in your schema as a reference to another collection is not checked as existing in the referenced collection on a save. You could do it in Mongoose middleware, if needed.
I found this thread very helpful and this is what I came up with:
This Middleware (I think its one anyway please let me know if not) I wrote checks the referenced model for the id provided in the field.
const mongoose = require('mongoose');
module.exports = (value, respond, modelName) => {
return modelName
.countDocuments({ _id: value })
.exec()
.then(function(count) {
return count > 0;
})
.catch(function(err) {
throw err;
});
};
Example model:
const mongoose = require('mongoose');
const uniqueValidator = require('mongoose-unique-validator');
const Schema = mongoose.Schema;
const User = require('./User');
const Cart = require('./Cart');
const refIsValid = require('../middleware/refIsValid');
const orderSchema = new Schema({
name: { type: String, default: Date.now, unique: true },
customerRef: { type: Schema.Types.ObjectId, required: true },
cartRef: { type: Schema.Types.ObjectId, ref: 'Cart', required: true },
total: { type: Number, default: 0 },
city: { type: String, required: true },
street: { type: String, required: true },
deliveryDate: { type: Date, required: true },
dateCreated: { type: Date, default: Date.now() },
ccLastDigits: { type: String, required: true },
});
orderSchema.path('customerRef').validate((value, respond) => {
return refIsValid(value, respond, User);
}, 'Invalid customerRef.');
orderSchema.path('cartRef').validate((value, respond) => {
return refIsValid(value, respond, Cart);
}, 'Invalid cartRef.');
orderSchema.path('ccLastDigits').validate(function(field) {
return field && field.length === 4;
}, 'Invalid ccLastDigits: must be 4 characters');
orderSchema.plugin(uniqueValidator);
module.exports = mongoose.model('order', orderSchema);
I'm a very new dev so any feedback is greatly valued!
You can try https://www.npmjs.com/package/lackey-mongoose-ref-validator (I'm the developer)
It also prevents deletion if the reference is used on another document.
var mongooseRefValidator = require('lackey-mongoose-ref-validator');
mongoSchema.plugin(mongooseRefValidator, {
onDeleteRestrict: ['tags']
});
It's an early version, so some bugs are expected. Just fill in a ticket if you find any.
I know this is an old thread but I had the same problem and I came up with a more "modern" solution.
I'm not an expert myself, hope I'm not misleading anyone, but this seems to work:
for example, in a simple "notes" schema, which contains a user field:
const noteSchema = new Schema({
user: { type: Schema.Types.ObjectId, ref: 'User' },
text: String
});
here's the middleware that checks if the userId exists:
noteSchema.path('user').validate(async (value) => {
return await User.findById(value);
}, 'User does not exist');