How do Functions work in a Mongoose Schema? - mongodb

I have a schema where I'm trying to generate a nanoid to show on the front end. (The default MongoDB ObjectID is too long to display to users.)
What I've done is to insert it into my schema as a default value that generates a new string each time an instance of the model is created.
These are the instructions provided by the nanoid docs for dealing with Mongoose
const mySchema = new Schema({
_id: {
type: String,
default: () => nanoid()
}
})
I've replicated this in my code:
const MySchema = new Schema(
{
// a bunch of other properties
nano_id: {
type: String,
default: () => nanoid()
}
},
{ autoIndex: true },
)
The problem I've run into is that for whatever reason. When I use my UI to create a new model with this new code on the back end, after hitting the submit button, the state changes to loading and gets stuck there.
I found a little hack where I can save my code in VSCode which forces a hot reload. This breaks out of the loading stage where I can then resubmit and it works as expected - nanoid is generated, displayed every where I want it, etc.
When my new nano_id property is removed from the Schema, I don't have to do this reloading hack to get it to save. So I know the bug has something to do with this.
What is actually happening with the syntax default: () => nanoid()?
My hypothesis is that there is some promise that is being generated or something that isn't resolving properly. Can't seem to find anything about this in the docs though. Other than related specifically to Date and Date.now, which I'm already using and is working fine.
Edit: adding function code
// at top of file where Schema is defined
const { customAlphabet } = require('nanoid');
const alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
const nanoid = customAlphabet(alphabet, 8);
Edit: adding request handler
router.post("/create", async (req, res) => {
const payload = req.body
// Map request payload to Referral Model
const new_referral = new Referral({
referral_status: payload.referral_status,
referral_agency: payload.referral_agency,
date_assigned: payload.date_assigned,
referral_source: payload.referral_source,
referral_source_email: payload.referral_source_email,
referral_source_name: payload.referral_source_name,
referral_title: payload.referral_title,
referral_source_phone: payload.referral_source_phone,
referral_need: payload.referral_need || [],
additional_details: payload.additional_details,
dob: payload.dob,
last_name: payload.last_name,
first_name: payload.first_name,
middle_name: payload.middle_name,
gender: payload.gender,
language: payload.language,
address_1: payload.address_1,
address_2: payload.address_2,
city: payload.city,
state: payload.state,
county: payload.county,
zip: payload.zip,
mailing_address_1: payload.mailing_address_1,
mailing_address_2: payload.mailing_address_2,
mailing_city: payload.mailing_city,
mailing_state: payload.mailing_state,
mailing_zip: payload.mailing_zip,
phone: payload.phone,
primary_phone_valid: payload.primary_phone_valid,
phone_secondary: payload.phone_secondary,
secondary_phone_valid: payload.secondary_phone_valid,
leave_message: payload.leave_message,
insurance_1: payload.insurance_1,
insurance_1_plan: payload.insurance_1_plan,
insurance_1_group_number: payload.insurance_1_group_number,
insurance_1_policy_number: payload.insurance_1_policy_number,
insurance_2: payload.insurance_2,
insurance_2_plan: payload.insurance_2_plan,
insurance_2_group_number: payload.insurance_2_group_number,
insurance_2_policy_number: payload.insurance_2_policy_number,
insurance_3: payload.insurance_3,
insurance_3_plan: payload.insurance_3_plan,
insurance_3_group_number: payload.insurance_3_group_number,
insurance_3_policy_number: payload.insurance_3_policy_number,
appointment_availability: payload.appointment_availability,
preferred_times: payload.preferred_times,
appointment_date: payload.appointment_date,
appointment_time: payload.appointment_time,
assessment_harp: payload.assessment_harp,
assessment_contact_date: payload.assessment_contact_date,
assessment_date_completed: payload.assessment_date_completed,
assessment_date_submitted: payload.assessment_date_submitted,
assessment_date_approved: payload.assessment_date_approved,
cin: payload.cin,
number_cancelled: payload.number_cancelled,
number_noshows: payload.number_noshows,
date_received: payload.date_received,
history: payload.history,
consent: payload.consent,
consent_date: payload.date_received,
project: payload.project || null,
contact_needs: payload.contact_needs || [],
contact_type: payload.contact_type || [],
notes: payload.notes,
contact_details: payload.contact_details,
contact_date: payload.contact_date,
contact_time: payload.contact_time,
number_of_contacts: payload.number_of_contacts,
address_type: payload.address_type,
shelter_name: payload.shelter_name,
address_valid: payload.address_valid,
agency_therapist: payload.agency_therapist,
phone_agency_therapist: payload.phone_agency_therapist,
email_agency_therapist: payload.email_agency_therapist,
agency_prescribing_clinician: payload.agency_prescribing_clinician,
phone_agency_prescribing_clinician:
payload.phone_agency_prescribing_clinician,
email_agency_prescribing_clinician:
payload.email_agency_prescribing_clinician,
primary_care_provider: payload.primary_care_provider,
phone_primary_care_provider: payload.phone_primary_care_provider,
email_primary_care_provider: payload.email_primary_care_provider,
email_client: payload.email_client,
referral_encounter_note: payload.referral_encounter_note,
health_home_enrolled: payload.health_home_enrolled,
health_home_facility: payload.health_home_facility,
health_home_enrollment_date: payload.health_home_enrollment_date,
care_management_enrolled: payload.care_management_enrolled,
care_management_facility: payload.care_management_facility,
care_management_enrollment_date: payload.care_management_enrollment_date,
last_annual_physical_date: payload.last_annual_physical_date,
last_pcp_followup_date: payload.last_pcp_followup_date,
last_annual_physical_time: payload.last_annual_physical_time,
last_pcp_followup_time: payload.last_pcp_followup_time,
insurance_1_effective_date: payload.insurance_1_effective_date,
insurance_1_expiration_date: payload.insurance_1_expiration_date,
insurance_2_effective_date: payload.insurance_2_effective_date,
insurance_2_expiration_date: payload.insurance_2_expiration_date,
insurance_3_effective_date: payload.insurance_3_effective_date,
insurance_3_expiration_date: payload.insurance_3_expiration_date,
qm_1_checkbox: payload.qm_1_checkbox,
qm_1_date: payload.qm_1_date,
qm_2_checkbox: payload.qm_2_checkbox,
qm_2_date: payload.qm_2_date,
qm_3_checkbox: payload.qm_3_checkbox,
qm_3_date: payload.qm_3_date,
qm_4_checkbox: payload.qm_4_checkbox,
qm_4_date: payload.qm_4_date,
qm_5_checkbox: payload.qm_5_checkbox,
qm_5_date: payload.qm_5_date,
qm_6_checkbox: payload.qm_6_checkbox,
qm_6_date: payload.qm_6_date,
is_guardian: payload.is_guardian,
guardian_first_name: payload.guardian_first_name,
guardian_last_name: payload.guardian_last_name,
engagement_contact_type: payload.engagement_contact_type,
engagement_date: payload.engagement_date,
engagement_time: payload.engagement_time,
engagement_number_of_contacts: payload.engagement_number_of_contacts,
engagement_contact_details: payload.engagement_contact_details,
active_duty: payload.active_duty,
veteran: payload.veteran,
programs: payload.programs,
outcomes: payload.outcomes,
user_access: payload.user_access,
ethnicity: payload.ethnicity,
quality_metrics: payload.quality_metrics,
hospital_site_address: payload.hospital_site_address,
hospital_site_name: payload.hospital_site_name,
hospital_site_npi: payload.hospital_site_npi,
hospital_site_phone: payload.hospital_site_phone,
pcp_site_name: payload.pcp_site_name,
pcp_name: payload.pcp_name,
pcp_npi: payload.pcp_npi,
pcp_phone: payload.pcp_phone,
pcp_address: payload.pcp_address,
care_management: {
contact_needs: payload.care_management_contact_needs,
contact_type: payload.care_management_contact_type,
number_of_contacts: payload.care_management_number_of_contacts,
appointment_date: payload.care_management_appointment_date,
appointment_time: payload.care_management_appointment_time,
contact_date: payload.care_management_contact_date,
contact_time: payload.care_management_contact_time,
contact_details: payload.care_management_contact_details,
contact_duration: payload.care_management_contact_duration,
number_of_contacts: payload.care_management_number_of_contacts,
},
peer_services: {
outcome_of_intervention: payload.peer_services_outcome_of_intervention,
contact_needs: payload.peer_services_contact_needs,
contact_type: payload.peer_services_contact_type,
number_of_contacts: payload.peer_services_number_of_contacts,
appointment_date: payload.peer_services_appointment_date,
appointment_time: payload.peer_services_appointment_time,
contact_date: payload.peer_services_contact_date,
contact_time: payload.peer_services_contact_time,
contact_details: payload.peer_services_contact_details,
contact_duration: payload.peer_services_contact_duration,
number_of_contacts: payload.peer_services_number_of_contacts,
},
physical_health: {
contact_needs: payload.physical_health_contact_needs,
contact_type: payload.physical_health_contact_type,
number_of_contacts: payload.physical_health_number_of_contacts,
contact_date: payload.physical_health_contact_date,
contact_time: payload.physical_health_contact_time,
contact_details: payload.physical_health_contact_details,
contact_duration: payload.physical_health_contact_duration,
number_of_contacts: payload.physical_health_number_of_contacts,
hospital_site_address: payload.physical_health_hospital_site_address,
hospital_site_name: payload.physical_health_hospital_site_name,
hospital_site_npi: payload.physical_health_hospital_site_npi,
hospital_site_phone: payload.physical_health_hospital_site_phone,
pcp_site_name: payload.physical_health_pcp_site_name,
pcp_name: payload.physical_health_pcp_name,
pcp_npi: payload.physical_health_pcp_npi,
pcp_phone: payload.physical_health_pcp_phone,
pcp_address: payload.physical_health_pcp_address,
pcp_appointment_date: payload.physical_health_pcp_appointment_date,
pcp_appointment_time: payload.physical_health_pcp_appointment_time,
last_pcp_followup_date: payload.physical_health_last_pcp_followup_date,
last_pcp_followup_time: payload.physical_health_last_pcp_followup_time,
last_annual_physical_date:
payload.physical_health_last_annual_physical_date,
last_annual_physical_time:
payload.physical_health_last_annual_physical_time,
breast_cancer_screening: payload.breast_cancer_screening,
breast_cancer_screening_date: payload.breast_cancer_screening_date,
well_child_visit_3_18: payload.well_child_visit_3_18,
well_child_visit_3_18_date: payload.well_child_visit_3_18_date,
well_child_first_30: payload.well_child_first_30,
well_child_first_30_date: payload.well_child_first_30_date,
seven_day_hospital_follow_up: payload.seven_day_hospital_follow_up,
seven_day_hospital_follow_up_date:
payload.seven_day_hospital_follow_up_date,
thirty_day_hospital_follow_up: payload.thirty_day_hospital_follow_up,
thirty_day_hospital_follow_up_date:
payload.thirty_day_hospital_follow_up_date,
emergency_dept_follow_up: payload.emergency_dept_follow_up,
emergency_dept_follow_up_date: payload.emergency_dept_follow_up_date,
diabetes_screening_schizo_bipolar:
payload.diabetes_screening_schizo_bipolar,
diabetes_screening_schizo_bipolar_date:
payload.diabetes_screening_schizo_bipolar_date,
potential_avoidable_ed_utilization:
payload.potential_avoidable_ed_utilization,
potential_avoidable_ed_utilization_date:
payload.potential_avoidable_ed_utilization_date,
cervical_cancer_screening: payload.cervical_cancer_screening,
cervical_cancer_screening_date: payload.cervical_cancer_screening_date,
csc_eye: payload.csc_eye,
csc_eye_date: payload.csc_eye_date,
amr: payload.amr,
amr_date: payload.amr_date,
col: payload.col,
col_date: payload.col_date,
},
outpatient_mental_health: {
contact_needs: payload.outpatient_mental_contact_needs,
contact_type: payload.outpatient_mental_contact_type,
number_of_contacts: payload.outpatient_mental_number_of_contacts,
appointment_date: payload.outpatient_mental_appointment_date,
appointment_time: payload.outpatient_mental_appointment_time,
contact_date: payload.outpatient_mental_contact_date,
contact_time: payload.outpatient_mental_contact_time,
contact_details: payload.outpatient_mental_contact_details,
contact_duration: payload.outpatient_mental_contact_duration,
number_of_contacts: payload.outpatient_mental_number_of_contacts,
type_of_appointment: payload.outpatient_mental_type_of_appointment,
diagnosis: payload.outpatient_mental_diagnosis,
grpa: payload.outpatient_mental_grpa,
grpa_date_completed: payload.outpatient_mental_grpa_date_completed,
grpa_frequency: payload.outpatient_mental_grpa_frequency,
noms: payload.outpatient_mental_noms,
noms_date_completed: payload.outpatient_mental_noms_date_completed,
noms_frequency: payload.outpatient_mental_noms_frequency,
},
outpatient_substance_use: {
contact_needs: payload.outpatient_substance_contact_needs,
contact_type: payload.outpatient_substance_contact_type,
number_of_contacts: payload.outpatient_substance_number_of_contacts,
appointment_date: payload.outpatient_substance_appointment_date,
appointment_time: payload.outpatient_substance_appointment_time,
contact_date: payload.outpatient_substance_contact_date,
contact_time: payload.outpatient_substance_contact_time,
contact_details: payload.outpatient_substance_contact_details,
contact_duration: payload.outpatient_substance_contact_duration,
number_of_contacts: payload.outpatient_substance_number_of_contacts,
type_of_appointment: payload.outpatient_substance_type_of_appointment,
diagnosis: payload.outpatient_substance_diagnosis,
grpa: payload.outpatient_substance_grpa,
grpa_date_completed: payload.outpatient_substance_grpa_date_completed,
grpa_frequency: payload.outpatient_mental_grpa_frequency,
noms: payload.outpatient_substance_noms,
noms_date_completed: payload.outpatient_substance_noms_date_completed,
noms_frequency: payload.outpatient_mental_noms_frequency,
},
})
if (!payload.duplicate_skip) {
// Check if there is already a referral with same first name, last name and dob. If so return an error
await Referral.find(
{
$and: [
{
last_name: {
$regex: new RegExp(`^${payload.last_name}$`, "i"),
},
first_name: {
$regex: new RegExp(`^${payload.first_name}$`, "i"),
},
dob: {
$regex: new RegExp(`^${payload.dob}$`, "i"),
},
project: {
$regex: new RegExp(`^${payload.project}$`, "i"),
},
},
],
},
(err, response) => {
if (response.length !== 0) {
var message = {
message:
"Referral with the same First Name, Last Name and Date of Birth already exists.",
type: "duplicate",
}
duplicate = true
return res.status(400).json(message)
}
},
)
} else {
new_referral
.save()
.then((referral) => {
// If an agency adds a referral with 'Agency' field blank / None, send CBHS email that new referral was added
if (
referral.referral_agency === "None" ||
referral.referral_agency === "CBHS" ||
referral.referral_agency === ""
) {
const send_to = "CBHS"
updateAgencyNotification(send_to, referral.project[0])
}
return res.status(200).json(referral)
})
.catch((err) => {
// Validation error, missing keys on required fields for referral creation
logger.error(req.originalUrl + err)
if (err.name === "ValidationError") {
missing_keys = []
for (err in err.errors) {
switch (err) {
case "referral_source":
missing_keys.push("Referral Source")
break
case "referral_source_email":
missing_keys.push("Referral Source Email")
break
case "referral_need":
missing_keys.push("Referral Need")
break
case "last_name":
missing_keys.push("Last Name")
break
case "first_name":
missing_keys.push("First Name")
break
case "middle_name":
missing_keys.push("Middle Name")
break
case "gender":
missing_keys.push("Gender")
break
case "address_1":
missing_keys.push("Address 1")
break
case "county":
missing_keys.push("County")
break
case "zip":
missing_keys.push("Zipcode")
break
case "phone":
missing_keys.push("Phone Number")
break
}
}
const validationError = {
message: "Validation Error",
fields: missing_keys,
}
return res.status(400).json(validationError)
}
})
}
})

problem here was there was no logic to save when duplicate skip was false.
if (!payload.duplicate_skip)
so it looked for something with the same first, last and dob,
await Referral.find(
{
$and: [
{
last_name: {
$regex: new RegExp(`^${payload.last_name}$`, "i"),
},
first_name: {
$regex: new RegExp(`^${payload.first_name}$`, "i"),
},
dob: {
$regex: new RegExp(`^${payload.dob}$`, "i"),
},
project: {
$regex: new RegExp(`^${payload.project}$`, "i"),
},
},
],
}
and if there was no match, it just did nothing.
the logic to save
else {
new_referral
.save()
.then((referral) => {
// If an agency adds a referral with 'Agency' field blank / None, send CBHS email that new referral was added
if (
referral.referral_agency === "None" ||
referral.referral_agency === "CBHS" ||
referral.referral_agency === ""
) {
const send_to = "CBHS"
updateAgencyNotification(send_to, referral.project[0])
}
return res.status(200).json(referral)
})
.catch((err) => {
// Validation error, missing keys on required fields for referral creation
logger.error(req.originalUrl + err)
if (err.name === "ValidationError") {
missing_keys = []
for (err in err.errors) {
switch (err) {
case "referral_source":
missing_keys.push("Referral Source")
break
case "referral_source_email":
missing_keys.push("Referral Source Email")
break
case "referral_need":
missing_keys.push("Referral Need")
break
case "last_name":
missing_keys.push("Last Name")
break
case "first_name":
missing_keys.push("First Name")
break
case "middle_name":
missing_keys.push("Middle Name")
break
case "gender":
missing_keys.push("Gender")
break
case "address_1":
missing_keys.push("Address 1")
break
case "county":
missing_keys.push("County")
break
case "zip":
missing_keys.push("Zipcode")
break
case "phone":
missing_keys.push("Phone Number")
break
}
}
const validationError = {
message: "Validation Error",
fields: missing_keys,
}
return res.status(400).json(validationError)
}
})
}
was never triggered when skip was false
so i added it after
if (response.length !== 0) {
var message = {
message:
"Referral with the same First Name, Last Name and Date of Birth already exists.",
type: "duplicate",
}
duplicate = true
return res.status(400).json(message)
}
meaning, "after checking for a duplicate (skip is false), if you don't get anything back (response === 0), then go ahead and actually create and save a new instance of the model"

Related

MongoDB query with 300k documents takes more than 30 seconds

Ok, as said in title, I have "performance issue" where I need to get all documents from a collection but it takes too long. Players collection contains around 300k documents with small size and query in service goes like this:
async getAllPlayers() {
const players = await this.playersCollection.find({}, {projection: { playerId: 1, name: 1, surname: 1, shirtNumber: 1, position: 1 }}).toArray();
return players;
}
Overall size is 6.4MB. I'm using Fastify adapter, fastify-compress and mongodb native driver. If I remove projection, it takes almost a minute.
Any idea how to improve this?
The best time I get is 8 seconds, where fast-json-stringify give me more than 10 seconds boost over 300k records:
'use strict'
// run fresh mongo
// docker run --name temp --rm -p 27017:27017 mongo
const fastify = require('fastify')({ logger: true })
const fjs = require('fast-json-stringify')
const toString = fjs({
type: 'object',
properties: {
playerId: { type: 'integer' },
name: { type: 'string' },
surname: { type: 'string' },
shirtNumber: { type: 'integer' },
}
})
fastify.register(require('fastify-mongodb'), {
forceClose: true,
url: 'mongodb://localhost/mydb'
})
fastify.get('/', (request, reply) => {
const dataStream = fastify.mongo.db.collection('foo')
.find({}, {
limit: 300000,
projection: { playerId: 1, name: 1, surname: 1, shirtNumber: 1, position: 1 }
})
.stream({
transform(doc) {
return toString(doc) + '\n'
}
})
reply.type('application/jsonl')
reply.send(dataStream)
})
fastify.get('/insert', async (request, reply) => {
const collection = fastify.mongo.db.collection('foo')
const batch = collection.initializeOrderedBulkOp();
for (let i = 0; i < 300000; i++) {
const player = {
playerId: i,
name: `Name ${i}`,
surname: `surname ${i}`,
shirtNumber: i
}
batch.insert(player);
}
const { result } = await batch.execute()
return result
})
fastify.listen(8080)
In any case, you should consider to:
paginate your output
or pushing the data into a bucket (like S3) and return to the client a URL to download the file directly, this will speed up a lot the process and will save your node.js process from this data streaming
Note that the compression in node.js is a heavy process, so it slows it down a lot the response. An nginx proxy adds it by default without the need to implement it in your business logic server.

Variables exporting error in MongoDB error

Question
I have provided my code below for reference. I'm using MongoDB and discord.js v12. So basically, I have made a !info command which shows some general info of the user.
What this code does is, it checks through the member's roles, and regarding which role they have, it calculates their total claim time (for giveaways etc.). The problem here, is with the donator role. I can't figure out why I can't use the donates variable outside the db.findOne block. Here, data.content.length shows the total donates of the users, which means donates * 5 is +5 claim time for each donate.
My Code
const moment = require('moment');
module.exports = {
name: 'info',
async execute(client, message, args, Discord){
const member = message.mentions.members.first() || message.guild.members.cache.get(args[0]) || message.member;
const db = require('../models/d-schema');
db.findOne({ guildid: message.guild.id, user: member.user.id }, async(err, data)=>{
if(err) throw err;
if(data){
const donates = parseInt(data.content.length);
}
})
var DefaultTime = 10;
var support = 0;
var donate = 0;
var boost = 0;
const userRoles = member.roles.cache.map((r) => r.name);
if (userRoles.includes("୨・supporter")) {
support = 3;
}
if (userRoles.includes("୨・donator")) {
donate = donates * 5;
}
if (userRoles.includes("୨・booster")) {
boost = 10;
}
const TotalTime = DefaultTime + support + donate + boost;
const embed = new Discord.MessageEmbed()
.setThumbnail(member.user.displayAvatarURL( {dynamic: true} ))
.addFields(
{name: member.user.tag, value: member.user, inline: true},
{name: 'Nickname', value: `${member.nickname !== null ? member.nickname : 'None'}`, inline: true},
{name: 'Is Bot', value: member.user.bot, inline: true},
{name: 'Joined', value: `${moment.utc(member.joinedAt).format("MMMM Do YYYY")}`, inline: true},
{name: 'Created', value: `${moment.utc(member.user.createdAt).format("MMMM Do YYYY")}`, inline: true},
{name: 'Claim Time', value: `${TotalTime} seconds`, inline: true},
)
.setFooter(`ID : ${member.user.id}`)
.setTimestamp()
.setColor('00ffcc')
message.channel.send(embed)
}
}
You cannot use the donates variable because you are declaring it inside the db.findOne() block. This is called variables scope. For better understanding you can read this answer.
If you want to use it outside of that block, you have to declare it beforehand, like this:
let donates;
db.findOne({ guildid: message.guild.id, user: member.user.id }, async(err, data)=>{
if(err) throw err;
if(data){
donates = parseInt(data.content.length);
}
})
Now you're able to use that variable outside of the db.findOne() block :)
Edit:
Alternative way:
It would be easier to use the function asynchronously. That way, everything can be scoped in the same block!
Example:
These two methods will give the same results:
const data = await Model.findOne({ ... });
console.log(data);
Model.findOne({ ... }, (err, data) => {
console.log(data);
});
Suggestion from Lioness100

Meteor Mongo Collections find forEach cursor iteration and saving to ElasticSearch Problem

i have Meteor App which is connected to MongoDB.
In mongo i have a table which has ~700k records.
I have a cron job each week, where i read all the records from the table (using Mongo Cursor) and in batches of 10k i want to insert them inside Elastic Search so they are indexed.
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Since for each is synchronous, goes over each record before it continues, and client.bulk is async, this is overloading the elastic search server and it crashes with Out of Memory Exception.
Is there a way to pause the forEach during the time when the insert is being done? I tried async/await but this does not seem to work as well.
let articles = []
Collections.Articles.find({}).forEach(async function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
await client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Any way how to achieve this?
EDIT: I am trying to achieve something like this - if i use promises
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
// Pause FETCHING rows with forEach
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles }).then(() => {
console.log('inserted')
// RESUME FETCHING rows with forEach
console.log("RESUME READING");
})
data = []
}
})
Managed to get this working with ES2018 Async iteration
Got an idea from
Using async/await with a forEach loop
Here is the code that is working
let articles = []
let cursor = Collections.Articles.find({})
for await (doc of cursor) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (articles.length === 10000) {
await client.bulk({ maxRetries: 5, index: 'trusted', type: 'artikel', body: articles })
articles = []
}
}
This works correctly and it manages to insert all the records into Elastic Search without crashing.
If you are concerned with the unthrottled iteration, then may use the internal Meteor._sleepForMs method, that allows you to put a async timeout in your sync-styled code:
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
Now this works fine within the Meteor environment (Meteor.startup, Meteor.methods, Meteor.publish).
You cron is likely to be not within this environment (= Fiber) so you may write a wrapper that binds the environment:
const bound = fct => Meteor.bindEnvironment(fct)
const iterateSlow = bound(function (timeout) {
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
iterateSlow(50) // iterates with 50ms timeout
Here is a complete minimal example, that you can reproduce with a fresh project:
// create a minimal collection
const MyDocs = new Mongo.Collection('myDocs')
// fill the collection
Meteor.startup(() => {
for (let i = 0; i < 100; i++) {
MyDocs.insert({})
}
})
// bind helper
const bound = fct => Meteor.bindEnvironment(fct)
// iterate docs with interval between
const iterateSlow = bound(function (timeout) {
MyDocs.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
// simulate external environment, like when cron runs
setTimeout(() => {
iterateSlow(50)
}, 2000)

Where to find Mongo Collection in Express?

I'm reviewing some old code I wrote in Express/Mongo/Mongoose (based on an online tutorial) and can't locate the portion of code which dictates which Collection in MongoDB gets written to.
I have a Database, UsersDB and within that database there are several Collections. The Collection that keeps growing every time the code executes user.save is the users Collection. But I can't find any reference in my code base to the users collection.
The only place in my code where I save a User is:
var app = require('../app');
var util = require('util');
var User = require('../models/user'),
Auth = User.Auth,
Name = User.Name,
Email= User.Email,
Phone = User.Phone,
Address = User.Address,
Company = User.Company,
PersonalData = User.PersonalData,
Id = User.Id,
Photo = User.Photo,
Member = User.Member,
CreditCard = User.CreditCard,
UserObj = User.User;
var moment = require('moment');
var async = require('async');
. . .
. . .
exports.user_create_post = [
(req,res, next) => {
console.log("Request: " + util.inspect(req.body));
},
//VALIDATE
body('mainEmail', 'Must be valid email.').isLength({min: 5}).trim(),
//SANITIZE
sanitizeBody('*').escape(),
//POPULATE NEW DOCUMENT
(req,res,next) => {
const errors = validationResult(req);
var auth = new Auth(
dateEffective: {value: moment(Date.now()).format("YYYY-MM-DD hh:mm:ss SSS"), attr: {hidden: true, label: ""}},
username: {"value": req.body.username, "attr": {hidden: false, label: "Username: "}},
password: {"value": req.body.password, "attr": {hidden: false, label: "Password: "}},
mainEmail: {"value": req.body.mainEmail, "attr": {hidden: false, label: "Email: "}}
});
var user = new UserObj(
{authData: [auth]}
);
if (!errors.isEmpty()) {
const errorFormatter = ({ location, msg, param, value, nestedErrors }) => {
// Build your resulting errors however you want! String, object, whatever - it works!
return `${location}[${param}]: ${msg}`;
};
const result = validationResult(req).formatWith(errorFormatter);
if (!result.isEmpty()) {
return res.json({ errors: result.array() });
}
}
else {
user.save(function(err){
if (err) { return next(err);}
});
res.redirect("http://localhost:4200/two-fa/"+user._id);
}
}
I also have a Models module (user.js):
. . .
. . .
. . .
module.exports = {
Auth: mongoose.model('Auth', AuthSchema),
Name: mongoose.model('Name', NameSchema),
Email: mongoose.model('Email', EmailSchema),
Phone: mongoose.model('Phone', PhoneSchema),
Address: mongoose.model('Address', AddressSchema),
Company: mongoose.model('Company', CompanySchema),
PersonalData: mongoose.model('PersonalData', PersonalDataSchema),
Id: mongoose.model('Id', IdSchema),
Photo: mongoose.model('Photo', PhoneSchema),
Member: mongoose.model('Member', MemberSchema),
CreditCard: mongoose.model('CreditCard', CreditCardSchema),
User: mongoose.model('User', UserSchema)
}
I did a search on my entire code, and nowhere is there any mention of users, which is the Collection that's getting written to.
Where should I look to try to trace how the users collection is getting written to?
Thank you!
As per how-to-access-a-preexisting-collection-with-mongoose, if a 3rd argument to mongoose.model is not provided, then Mongo automatically "pluralizes" the model name to give the name of the Collection.

DynamoDB PutItemOutcome#getPutItemResult() returns empty object

I'm trying to run the following example from AWS Dynamo tutorial locally, Step 3: Put, Update, and Delete an Item.
In my case, it is:
val client: AmazonDynamoDBClient = new AmazonDynamoDBClient().withEndpoint("http://localhost:7777")
val dynamoDB: DynamoDB = new DynamoDB(client)
val table: Table = dynamoDB.getTable("Catalog")
try {
val rating: java.util.List[Float] = new java.util.LinkedList[Float]()
rating.add(1)
val newItem: Item = new Item().withPrimaryKey("Title", "Title here").withInt("Country", 1).
withList("Ratings", rating)
val outcome: PutItemOutcome = table.putItem(newItem)
System.out.println("PutItem succeeded:\n" + outcome.getPutItemResult)
} catch {
case exception: Exception => System.out.println(exception.getMessage)
}
The output is:
PutItem succeeded: {}
While in local DynamoDB console:
var params = {
TableName: "Catalog",
Key: {
"Title":"Title Here",
}
};
docClient.get(params, function(err, data) {
if (err)
console.log(JSON.stringify(err, null, 2));
else
console.log(JSON.stringify(data, null, 2));
});
Output:
{
"Item": {
"Title": "Title Here",
"Ratings": [
1
],
"Country": 1
}
}
You need to set ReturnValues to ALL_OLD in the PutItem request to get a value returned, but even then it will only have the values that were replaced.
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html#DDB-PutItem-request-ReturnValues
With your code you'd need to do something like replace
val outcome: PutItemOutcome = table.putItem(newItem)
with
val putItemSpec: PutItemSpec = new PutItemSpec()
.withItem(newItem)
.withReturnValues(ReturnValue.ALL_OLD)
val outcome: PutItemOutcome = table.putItem(putItemSpec)