Correct way to seed MongoDB with references via mongoose - mongodb

I have three schemas, one which references two others:
userSchema
{ name: String }
postSchema
{ content: String }
commentSchema
{
content: String,
user: { ObjectID, ref: 'User' },
post: { ObjectID, ref: 'Post' }
}
How can I seed this database in a sane, scalable way? Even using bluebird promises it quickly becomes a nightmare to write.
My attempt so far involves multiple nested promises and is very hard to maintain:
User
.create([{ name: 'alice' }])
.then(() => {
return Post.create([{ content: 'foo' }])
})
.then(() => {
User.find().then(users => {
Post.find().then(posts => {
// `users` isn't even *available* here!
Comment.create({ content: 'bar', user: users[0], post: posts[0] })
})
})
})
This is clearly not the correct way of doing this. What am I missing?

Not sure about bluebird, but the nodejs Promise.all should do the job:
Promise.all([
User.create([{ name: 'alice' }]),
Post.create([{ content: 'foo' }])
]).then(([users, posts]) => {
const comments = [
{ content: 'bar', user: users[0], post: posts[0] }
];
return Comment.create(comments);
})

If you want to seed database with automatically references, use Seedgoose.
This is the easiest seeder for you to use. You don't need to write any program files, but only data files. And Seedgoose handles smart references for you. And by the way, I'm the author and maintainer of this package.

Try this it will work fine:
Note: Node Promise.all will make sure that the both query is executed properly and then return the result in Array:[Users, Posts],
If you get any error during execution of any query, it will be handle by catch block of the Promise.all.
let queryArray = [];
queryArray.push(User.create([{ name: 'alice' }]));
queryArray.push(Post.create([{ content: 'foo' }]));
Promise.all(queryArray).then(([Users, Posts]) => {
const comments = [
{ content: 'bar', user: Users[0], post: posts[0] }
];
return Comment.create(comments);
}).catch(Error => {
console.log("Error: ", Error);
})

Related

Redux Toolkit Query: Reduce state from "mutation" response

Let's say I have an RESTish API to manage "posts".
GET /posts returns all posts
PATCH /posts:id updates a post and responds with new record data
I can implement this using RTK query via something like this:
const TAG_TYPE = 'POST';
// Define a service using a base URL and expected endpoints
export const postsApi = createApi({
reducerPath: 'postsApi',
tagTypes: [TAG_TYPE],
baseQuery,
endpoints: (builder) => ({
getPosts: builder.query<Form[], string>({
query: () => `/posts`,
providesTags: (result) =>
[
{ type: TAG_TYPE, id: 'LIST' },
],
}),
updatePost: builder.mutation<any, { formId: string; formData: any }>({
// note: an optional `queryFn` may be used in place of `query`
query: (data) => ({
url: `/post/${data.formId}`,
method: 'PATCH',
body: data.formData,
}),
// this causes a full re-query.
// Would be more efficient to update state based on resp.body
invalidatesTags: [{ type: TAG_TYPE, id: 'LIST' }],
}),
}),
});
When updatePost runs, it invalidates the LIST tag which causes getPosts to run again.
However, since the PATCH operation responds with the new data itself, I would like to avoid making an additional server request and instead just update my reducer state for that specific record with the content of response.body.
Seems like a common use case, but I'm struggling to find any documentation on doing something like this.
You can apply the mechanism described in optimistic updates, just a little bit later:
import { createApi, fetchBaseQuery } from '#reduxjs/toolkit/query'
import { Post } from './types'
const api = createApi({
// ...
endpoints: (build) => ({
// ...
updatePost: build.mutation<void, Pick<Post, 'id'> & Partial<Post>>({
query: ({ id, ...patch }) => ({
// ...
}),
async onQueryStarted({ id, ...patch }, { dispatch, queryFulfilled }) {
const { data } = await queryFulfilled
dispatch(
api.util.updateQueryData('getPost', id, (draft) => {
Object.assign(draft, data)
})
)
},
}),
}),
})

Meteor Mongo Collections find forEach cursor iteration and saving to ElasticSearch Problem

i have Meteor App which is connected to MongoDB.
In mongo i have a table which has ~700k records.
I have a cron job each week, where i read all the records from the table (using Mongo Cursor) and in batches of 10k i want to insert them inside Elastic Search so they are indexed.
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Since for each is synchronous, goes over each record before it continues, and client.bulk is async, this is overloading the elastic search server and it crashes with Out of Memory Exception.
Is there a way to pause the forEach during the time when the insert is being done? I tried async/await but this does not seem to work as well.
let articles = []
Collections.Articles.find({}).forEach(async function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
await client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Any way how to achieve this?
EDIT: I am trying to achieve something like this - if i use promises
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
// Pause FETCHING rows with forEach
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles }).then(() => {
console.log('inserted')
// RESUME FETCHING rows with forEach
console.log("RESUME READING");
})
data = []
}
})
Managed to get this working with ES2018 Async iteration
Got an idea from
Using async/await with a forEach loop
Here is the code that is working
let articles = []
let cursor = Collections.Articles.find({})
for await (doc of cursor) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (articles.length === 10000) {
await client.bulk({ maxRetries: 5, index: 'trusted', type: 'artikel', body: articles })
articles = []
}
}
This works correctly and it manages to insert all the records into Elastic Search without crashing.
If you are concerned with the unthrottled iteration, then may use the internal Meteor._sleepForMs method, that allows you to put a async timeout in your sync-styled code:
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
Now this works fine within the Meteor environment (Meteor.startup, Meteor.methods, Meteor.publish).
You cron is likely to be not within this environment (= Fiber) so you may write a wrapper that binds the environment:
const bound = fct => Meteor.bindEnvironment(fct)
const iterateSlow = bound(function (timeout) {
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
iterateSlow(50) // iterates with 50ms timeout
Here is a complete minimal example, that you can reproduce with a fresh project:
// create a minimal collection
const MyDocs = new Mongo.Collection('myDocs')
// fill the collection
Meteor.startup(() => {
for (let i = 0; i < 100; i++) {
MyDocs.insert({})
}
})
// bind helper
const bound = fct => Meteor.bindEnvironment(fct)
// iterate docs with interval between
const iterateSlow = bound(function (timeout) {
MyDocs.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
// simulate external environment, like when cron runs
setTimeout(() => {
iterateSlow(50)
}, 2000)

How can I retrieve an id from MongoDB create operation during a transaction?

I am trying to create an audit trail using Apollo Server and Mongoose. When a user initially registers, I create a document in the users collection and a document in the history collection for each piece of data they provided (username, password, email, etc) . For each history collection document, I include the id for the user document to create a relationship. Works perfectly.
However, when I add a transaction in (see below), the userId for the user document comes back as undefined, so I cannot add it to the history entry documents. I am assuming that the id for a document does not get created until the entire transaction has been completed?
Any ideas?
Mutation: {
register: async (_, { data }) => {
// Start a mongo session & transaction
const session = await mongoose.startSession()
session.startTransaction()
try {
// Hash password and create user
const hashedPassword = await bcrypt.hash(data.password, 12)
const user = await User.create(
[{ ...data, password: hashedPassword }],
{ session }
)
// Add history entries
HistoryEntry.create([
{
user: user.id,
action: 'registered'
},
{
user: user.id,
action: 'set',
object: 'profile',
instance: user.id,
property: 'firstName',
value: firstName
},
{
user: user.id,
action: 'set',
object: 'profile',
instance: user.id,
property: 'lastName',
value: lastName
},
{
user: user.id,
action: 'set',
object: 'profile',
instance: user.id,
property: 'password'
}
])
if (loginType === 'email') {
HistoryEntry.create({
user: user.id,
action: 'set',
object: 'profile',
instance: user.id,
property: 'email',
value: login
})
}
if (loginType === 'mobile') {
HistoryEntry.create({
user: user.id,
action: 'set',
object: 'profile',
instance: user.id,
property: 'mobile',
value: login
})
}
// commit the changes if everything was successful
await session.commitTransaction()
return {
ok: true,
user
}
} catch (err) {
// if anything fails above, rollback the changes in the transaction
await session.abortTransaction()
return formatErrors(err)
} finally {
// end the session
session.endSession()
}
}
}
If you think about it, how can you add a HistoryEntry if you haven't added User yet? It's not a 'history' as you are currently doing it. I believe you got two options here - set _id on User manually new Schema({ _id: { type: Schema.ObjectId, auto: true }}) and then generate it within the transaction: var userId = ObjectId(); and use for both User and History Entries.
And the second option, more semantically correct in this context, I believe - you should attach to post-save hook:
schema.post('save', function(doc) {
console.log('%s has been saved', doc._id);
});
So, whenever an User is created, a post-save hook is fired to update History.
Came across the same issue recently, hope you have figured it out already. I may add this for future seekers.
Following create function returns an array of created documents.
const user = await User.create(
[{ ...data, password: hashedPassword }],
{ session }
);
Therefore access the user id as user[0]._id
Pass the session also to HistoryEntry.create()
HistoryEntry.create([{...},{...}], {session})
Note: In this use case, I personally prefer #marek second option to use a post-save hook.

How to make Mongoose model.insertMany insert documents with numerical and ordered ids?

I have this route in the backend express server:
router.route('/fillInformationAssetsSeverityEvaluation').post((req, res) => {
informationAssetsSeverityEvaluationRow.remove({}, (err) => {
if (err)
console.log(err);
else
// res.json("informationAssets Collection has been dropped!");
res.json('information Assets Severity Evaluation data has been received on the server side')
informationAssetsSeverityEvaluationRow.insertMany([req.body[0]], {
multi: true
}).then(documentsInserted => {
console.log('[req.body[0]]: ', [req.body[0]]);
console.log('documentsInserted: ', documentsInserted);
console.log('You have succesfully inserted ', documentsInserted.length, ' documents in informationAssetsSeverityEvaluation collection');
});
});
})
For the sake of simplicity, I am inserting only one document.
[req.body[0]]
{ REF: 'REFSHIT',
confFin: 'A',
confRep: 'A'}
But, in the real applications, I am inserting multiple documents similar to that.
This consoleLog :
console.log('documentsInserted: ', documentsInserted);
logs:
documentsInserted: [ { _id: 5d3453afc302d718e4870b53,
REF: 'REFSHIT',
confFin: 'A',
confRep: 'A'}]
As you see the id is automatically generated:
> _id: 5d3453afc302d718e4870b53
What I would like is: The ids of the different documents to be "numerically ordered". I.e:
Document 0 would have id 0
Document 1 would have id 1
Document 2 would have id 2
And so on and so forth.
After having made some research, I found out that I can do this manually by inserting the id manually inside the updateMany objects.
However, since I receive the documents objects from the request body, this is not a viable solution.
Any help?
Finally after trying four modules and a couple of days of trying for something that should be native to mongodb, I have found a simple solution. I hope it helps someone.
1/ Install mongoose-plugin-autoinc
2/
import mongoose from 'mongoose';
import { autoIncrement } from 'mongoose-plugin-autoinc';
const connection = mongoose.createConnection("mongodb://localhost/myDatabase");
const BookSchema = new mongoose.Schema({
author: { type: Schema.Types.ObjectId, ref: 'Author' },
title: String,
genre: String,
publishDate: Date
});
BookSchema.plugin(autoIncrement, 'Book');
const Book = connection.model('Book', BookSchema);
2/ In my case I have the models defined in models.js and the connection defined in server.js so I had to write this :
BookSchema.plugin(autoIncrement, 'Book');
in models.js
and instead of
const Book = connection.model('Book', BookSchema);
I have:
module.exports = {
informationAssetsRow: mongoose.model('informationAssetsRow', informationAssetsRow),
};
And in server.js:
const {
informationAssetsRow,
} = require('./models/models')

Asynchronous Issues with JEST and MongoDB

I am getting inconsistent results with JEST when I try to remove items from a MongoDB Collection using the beforeEach() Hook.
My Mongoose schema and model defined as:
// Define Mongoose wafer sort schema
const waferSchema = new mongoose.Schema({
productType: {
type: String,
required: true,
enum: ['A', 'B'],
},
updated: {
type: Date,
default: Date.now,
index: true,
},
waferId: {
type: String,
required: true,
trim: true,
minlength: 7,
},
sublotId: {
type: String,
required: true,
trim: true,
minlength: 7,
},
}
// Define unique key for the schema
const Wafer = mongoose.model('Wafer', waferSchema);
module.exports.Wafer = Wafer;
My JEST tests:
describe('API: /WT', () => {
// Happy Path for Posting Object
let wtEntry = {};
beforeEach(async () => {
wtEntry = {
productType: 'A',
waferId: 'A01A001.3',
sublotId: 'A01A001.1',
};
await Wafer.deleteMany({});
// I also tried to pass in done and then call done() after the delete
});
describe('GET /:id', () => {
it('Return Wafer Sort Entry with specified ID', async () => {
// Create a new wafer Entry and Save it to the DB
const wafer = new Wafer(wtEntry);
await wafer.save();
const res = await request(apiServer).get(`/WT/${wafer.id}`);
expect(res.status).toBe(200);
expect(res.body).toHaveProperty('productType', 'A');
expect(res.body).toHaveProperty('waferId', 'A01A001.3');
expect(res.body).toHaveProperty('sublotId', 'A01A001.1');
});
}
So the error I always get is related to duplicate keys when I run my tests more than once:
MongoError: E11000 duplicate key error collection: promis_tests.promiswts index: waferId_1_sublotId_1 dup key: { : "A01A001.3", : "A01A001.1" }
But I do not understand how I can get this duplicate key error if the beforeEach() were firing properly. Am I trying to clear the collection improperly? I've tried passing in a done element to the before each callback and invoking it after delete command. I've also tried implementing the delete in beforeAll(), afterEach(), and afterAll() but still get inconsistent results. I'm pretty stumped on this one. I might just removed the schema key all together but I would like to understand what is going on here with the beforeEach(). Thanks in advance for any advice.
It might be because you are not actually using the promise API that mongoose has to offer. By default, mongooses functions like deleteMany() do not return a promise. You will have to call .exec() at the end of the function chain to return a promise e.g. await collection.deleteMany({}).exec(). So you are running into a race condition. deleteMany() also accepts a callback, so you could always wrap it in a promise. I would do something like this:
describe('API: /WT', () => {
// Happy Path for Posting Object
const wtEntry = {
productType: 'A',
waferId: 'A01A001.3',
sublotId: 'A01A001.1',
};
beforeEach(async () => {
await Wafer.deleteMany({}).exec();
});
describe('GET /:id', () => {
it('Return Wafer Sort Entry with specified ID', async () => {
expect.assertions(4);
// Create a new wafer Entry and Save it to the DB
const wafer = await Wafer.create(wtEntry);
const res = await request(apiServer).get(`/WT/${wafer.id}`);
expect(res.status).toBe(200);
expect(res.body).toHaveProperty('productType', 'A');
expect(res.body).toHaveProperty('waferId', 'A01A001.3');
expect(res.body).toHaveProperty('sublotId', 'A01A001.1');
});
}
Also, always expect the assertions with asynchronous code
https://jestjs.io/docs/en/asynchronous.html
You can read more about mongoose promises and query objects here
https://mongoosejs.com/docs/promises.html
Without deleting the schema index this seems to be the most reliable solution. Not 100% sure why it works over async await Wafer.deleteMany({});
beforeEach((done) => {
wtEntry = {
productType: 'A',
waferId: 'A01A001.3',
sublotId: 'A01A001.1',
};
mongoose.connection.collections.promiswts.drop(() => {
// Run the next test!
done();
});
});