DynamoDB PutItemOutcome#getPutItemResult() returns empty object - scala

I'm trying to run the following example from AWS Dynamo tutorial locally, Step 3: Put, Update, and Delete an Item.
In my case, it is:
val client: AmazonDynamoDBClient = new AmazonDynamoDBClient().withEndpoint("http://localhost:7777")
val dynamoDB: DynamoDB = new DynamoDB(client)
val table: Table = dynamoDB.getTable("Catalog")
try {
val rating: java.util.List[Float] = new java.util.LinkedList[Float]()
rating.add(1)
val newItem: Item = new Item().withPrimaryKey("Title", "Title here").withInt("Country", 1).
withList("Ratings", rating)
val outcome: PutItemOutcome = table.putItem(newItem)
System.out.println("PutItem succeeded:\n" + outcome.getPutItemResult)
} catch {
case exception: Exception => System.out.println(exception.getMessage)
}
The output is:
PutItem succeeded: {}
While in local DynamoDB console:
var params = {
TableName: "Catalog",
Key: {
"Title":"Title Here",
}
};
docClient.get(params, function(err, data) {
if (err)
console.log(JSON.stringify(err, null, 2));
else
console.log(JSON.stringify(data, null, 2));
});
Output:
{
"Item": {
"Title": "Title Here",
"Ratings": [
1
],
"Country": 1
}
}

You need to set ReturnValues to ALL_OLD in the PutItem request to get a value returned, but even then it will only have the values that were replaced.
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html#DDB-PutItem-request-ReturnValues
With your code you'd need to do something like replace
val outcome: PutItemOutcome = table.putItem(newItem)
with
val putItemSpec: PutItemSpec = new PutItemSpec()
.withItem(newItem)
.withReturnValues(ReturnValue.ALL_OLD)
val outcome: PutItemOutcome = table.putItem(putItemSpec)

Related

use spark-mongo to upsert

my collection config
{ "_id" : "db_1.target_collection", "lastmodEpoch" :
ObjectId("6076a37e37c2cca5853da6df"), "lastmod" :
ISODate("1970-02-19T17:02:47.301Z"), "dropped" : false, "key" : {
"kfuin" : "hashed" }, "unique" : false, "uuid" :
UUID("57c30bbe-af83-4410-a51f-c04f3c7522f4") }
I want to read from mongo and update into mongo
df.write.format('com.mongodb.spark.sql') \
.option('collection', 'target_collection') \
.option('replaceDocument', 'false') \
.option('shardKey', '{kfuin: 1}') \
.mode('append') \
.save()
I am getting this exception when I try to upsert when replaceDocument is true
com.mongodb.MongoBulkWriteException: Bulk write operation error on
server ... message='After applying the update, the (immutable) field
'_id' was found to have been altered to _id:
ObjectId('5f80331981f3601291e04a1c')
and when replaceDocument is false
Performing an update on the path '_id' would modify the immutable
field '_id''.
Any ideas?
Resolved
if you want to update the data, shardKey need to add "_id"
.option('shardKey', '{kfuin: 1,_id: 1}')
MongoSpark.save method:
def save[D](dataset: Dataset[D], writeConfig: WriteConfig): Unit = {
val mongoConnector = MongoConnector(writeConfig.asOptions)
val dataSet = dataset.toDF()
val mapper = rowToDocumentMapper(dataSet.schema)
val documentRdd: RDD[BsonDocument] = dataSet.rdd.map(row => mapper(row))
val fieldNames = dataset.schema.fieldNames.toList
val queryKeyList = BsonDocument.parse(writeConfig.shardKey.getOrElse("{_id: 1}")).keySet().asScala.toList
if (writeConfig.forceInsert || !queryKeyList.forall(fieldNames.contains(_))) {
MongoSpark.save(documentRdd, writeConfig)
} else {
documentRdd.foreachPartition(iter => if (iter.nonEmpty) {
mongoConnector.withCollectionDo(writeConfig, { collection: MongoCollection[BsonDocument] =>
iter.grouped(writeConfig.maxBatchSize).foreach(batch => {
val requests = batch.map(doc =>
if (queryKeyList.forall(doc.containsKey(_))) {
val queryDocument = new BsonDocument()
queryKeyList.foreach(key => queryDocument.append(key, doc.get(key)))
if (writeConfig.replaceDocument) {
new ReplaceOneModel[BsonDocument](queryDocument, doc, new ReplaceOptions().upsert(true))
} else {
queryDocument.keySet().asScala.foreach(doc.remove(_))
new UpdateOneModel[BsonDocument](queryDocument, new BsonDocument("$set", doc), new UpdateOptions().upsert(true))
}
} else {
new InsertOneModel[BsonDocument](doc)
})
collection.bulkWrite(requests.toList.asJava)
})
})
})
}
}

How do Functions work in a Mongoose Schema?

I have a schema where I'm trying to generate a nanoid to show on the front end. (The default MongoDB ObjectID is too long to display to users.)
What I've done is to insert it into my schema as a default value that generates a new string each time an instance of the model is created.
These are the instructions provided by the nanoid docs for dealing with Mongoose
const mySchema = new Schema({
_id: {
type: String,
default: () => nanoid()
}
})
I've replicated this in my code:
const MySchema = new Schema(
{
// a bunch of other properties
nano_id: {
type: String,
default: () => nanoid()
}
},
{ autoIndex: true },
)
The problem I've run into is that for whatever reason. When I use my UI to create a new model with this new code on the back end, after hitting the submit button, the state changes to loading and gets stuck there.
I found a little hack where I can save my code in VSCode which forces a hot reload. This breaks out of the loading stage where I can then resubmit and it works as expected - nanoid is generated, displayed every where I want it, etc.
When my new nano_id property is removed from the Schema, I don't have to do this reloading hack to get it to save. So I know the bug has something to do with this.
What is actually happening with the syntax default: () => nanoid()?
My hypothesis is that there is some promise that is being generated or something that isn't resolving properly. Can't seem to find anything about this in the docs though. Other than related specifically to Date and Date.now, which I'm already using and is working fine.
Edit: adding function code
// at top of file where Schema is defined
const { customAlphabet } = require('nanoid');
const alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
const nanoid = customAlphabet(alphabet, 8);
Edit: adding request handler
router.post("/create", async (req, res) => {
const payload = req.body
// Map request payload to Referral Model
const new_referral = new Referral({
referral_status: payload.referral_status,
referral_agency: payload.referral_agency,
date_assigned: payload.date_assigned,
referral_source: payload.referral_source,
referral_source_email: payload.referral_source_email,
referral_source_name: payload.referral_source_name,
referral_title: payload.referral_title,
referral_source_phone: payload.referral_source_phone,
referral_need: payload.referral_need || [],
additional_details: payload.additional_details,
dob: payload.dob,
last_name: payload.last_name,
first_name: payload.first_name,
middle_name: payload.middle_name,
gender: payload.gender,
language: payload.language,
address_1: payload.address_1,
address_2: payload.address_2,
city: payload.city,
state: payload.state,
county: payload.county,
zip: payload.zip,
mailing_address_1: payload.mailing_address_1,
mailing_address_2: payload.mailing_address_2,
mailing_city: payload.mailing_city,
mailing_state: payload.mailing_state,
mailing_zip: payload.mailing_zip,
phone: payload.phone,
primary_phone_valid: payload.primary_phone_valid,
phone_secondary: payload.phone_secondary,
secondary_phone_valid: payload.secondary_phone_valid,
leave_message: payload.leave_message,
insurance_1: payload.insurance_1,
insurance_1_plan: payload.insurance_1_plan,
insurance_1_group_number: payload.insurance_1_group_number,
insurance_1_policy_number: payload.insurance_1_policy_number,
insurance_2: payload.insurance_2,
insurance_2_plan: payload.insurance_2_plan,
insurance_2_group_number: payload.insurance_2_group_number,
insurance_2_policy_number: payload.insurance_2_policy_number,
insurance_3: payload.insurance_3,
insurance_3_plan: payload.insurance_3_plan,
insurance_3_group_number: payload.insurance_3_group_number,
insurance_3_policy_number: payload.insurance_3_policy_number,
appointment_availability: payload.appointment_availability,
preferred_times: payload.preferred_times,
appointment_date: payload.appointment_date,
appointment_time: payload.appointment_time,
assessment_harp: payload.assessment_harp,
assessment_contact_date: payload.assessment_contact_date,
assessment_date_completed: payload.assessment_date_completed,
assessment_date_submitted: payload.assessment_date_submitted,
assessment_date_approved: payload.assessment_date_approved,
cin: payload.cin,
number_cancelled: payload.number_cancelled,
number_noshows: payload.number_noshows,
date_received: payload.date_received,
history: payload.history,
consent: payload.consent,
consent_date: payload.date_received,
project: payload.project || null,
contact_needs: payload.contact_needs || [],
contact_type: payload.contact_type || [],
notes: payload.notes,
contact_details: payload.contact_details,
contact_date: payload.contact_date,
contact_time: payload.contact_time,
number_of_contacts: payload.number_of_contacts,
address_type: payload.address_type,
shelter_name: payload.shelter_name,
address_valid: payload.address_valid,
agency_therapist: payload.agency_therapist,
phone_agency_therapist: payload.phone_agency_therapist,
email_agency_therapist: payload.email_agency_therapist,
agency_prescribing_clinician: payload.agency_prescribing_clinician,
phone_agency_prescribing_clinician:
payload.phone_agency_prescribing_clinician,
email_agency_prescribing_clinician:
payload.email_agency_prescribing_clinician,
primary_care_provider: payload.primary_care_provider,
phone_primary_care_provider: payload.phone_primary_care_provider,
email_primary_care_provider: payload.email_primary_care_provider,
email_client: payload.email_client,
referral_encounter_note: payload.referral_encounter_note,
health_home_enrolled: payload.health_home_enrolled,
health_home_facility: payload.health_home_facility,
health_home_enrollment_date: payload.health_home_enrollment_date,
care_management_enrolled: payload.care_management_enrolled,
care_management_facility: payload.care_management_facility,
care_management_enrollment_date: payload.care_management_enrollment_date,
last_annual_physical_date: payload.last_annual_physical_date,
last_pcp_followup_date: payload.last_pcp_followup_date,
last_annual_physical_time: payload.last_annual_physical_time,
last_pcp_followup_time: payload.last_pcp_followup_time,
insurance_1_effective_date: payload.insurance_1_effective_date,
insurance_1_expiration_date: payload.insurance_1_expiration_date,
insurance_2_effective_date: payload.insurance_2_effective_date,
insurance_2_expiration_date: payload.insurance_2_expiration_date,
insurance_3_effective_date: payload.insurance_3_effective_date,
insurance_3_expiration_date: payload.insurance_3_expiration_date,
qm_1_checkbox: payload.qm_1_checkbox,
qm_1_date: payload.qm_1_date,
qm_2_checkbox: payload.qm_2_checkbox,
qm_2_date: payload.qm_2_date,
qm_3_checkbox: payload.qm_3_checkbox,
qm_3_date: payload.qm_3_date,
qm_4_checkbox: payload.qm_4_checkbox,
qm_4_date: payload.qm_4_date,
qm_5_checkbox: payload.qm_5_checkbox,
qm_5_date: payload.qm_5_date,
qm_6_checkbox: payload.qm_6_checkbox,
qm_6_date: payload.qm_6_date,
is_guardian: payload.is_guardian,
guardian_first_name: payload.guardian_first_name,
guardian_last_name: payload.guardian_last_name,
engagement_contact_type: payload.engagement_contact_type,
engagement_date: payload.engagement_date,
engagement_time: payload.engagement_time,
engagement_number_of_contacts: payload.engagement_number_of_contacts,
engagement_contact_details: payload.engagement_contact_details,
active_duty: payload.active_duty,
veteran: payload.veteran,
programs: payload.programs,
outcomes: payload.outcomes,
user_access: payload.user_access,
ethnicity: payload.ethnicity,
quality_metrics: payload.quality_metrics,
hospital_site_address: payload.hospital_site_address,
hospital_site_name: payload.hospital_site_name,
hospital_site_npi: payload.hospital_site_npi,
hospital_site_phone: payload.hospital_site_phone,
pcp_site_name: payload.pcp_site_name,
pcp_name: payload.pcp_name,
pcp_npi: payload.pcp_npi,
pcp_phone: payload.pcp_phone,
pcp_address: payload.pcp_address,
care_management: {
contact_needs: payload.care_management_contact_needs,
contact_type: payload.care_management_contact_type,
number_of_contacts: payload.care_management_number_of_contacts,
appointment_date: payload.care_management_appointment_date,
appointment_time: payload.care_management_appointment_time,
contact_date: payload.care_management_contact_date,
contact_time: payload.care_management_contact_time,
contact_details: payload.care_management_contact_details,
contact_duration: payload.care_management_contact_duration,
number_of_contacts: payload.care_management_number_of_contacts,
},
peer_services: {
outcome_of_intervention: payload.peer_services_outcome_of_intervention,
contact_needs: payload.peer_services_contact_needs,
contact_type: payload.peer_services_contact_type,
number_of_contacts: payload.peer_services_number_of_contacts,
appointment_date: payload.peer_services_appointment_date,
appointment_time: payload.peer_services_appointment_time,
contact_date: payload.peer_services_contact_date,
contact_time: payload.peer_services_contact_time,
contact_details: payload.peer_services_contact_details,
contact_duration: payload.peer_services_contact_duration,
number_of_contacts: payload.peer_services_number_of_contacts,
},
physical_health: {
contact_needs: payload.physical_health_contact_needs,
contact_type: payload.physical_health_contact_type,
number_of_contacts: payload.physical_health_number_of_contacts,
contact_date: payload.physical_health_contact_date,
contact_time: payload.physical_health_contact_time,
contact_details: payload.physical_health_contact_details,
contact_duration: payload.physical_health_contact_duration,
number_of_contacts: payload.physical_health_number_of_contacts,
hospital_site_address: payload.physical_health_hospital_site_address,
hospital_site_name: payload.physical_health_hospital_site_name,
hospital_site_npi: payload.physical_health_hospital_site_npi,
hospital_site_phone: payload.physical_health_hospital_site_phone,
pcp_site_name: payload.physical_health_pcp_site_name,
pcp_name: payload.physical_health_pcp_name,
pcp_npi: payload.physical_health_pcp_npi,
pcp_phone: payload.physical_health_pcp_phone,
pcp_address: payload.physical_health_pcp_address,
pcp_appointment_date: payload.physical_health_pcp_appointment_date,
pcp_appointment_time: payload.physical_health_pcp_appointment_time,
last_pcp_followup_date: payload.physical_health_last_pcp_followup_date,
last_pcp_followup_time: payload.physical_health_last_pcp_followup_time,
last_annual_physical_date:
payload.physical_health_last_annual_physical_date,
last_annual_physical_time:
payload.physical_health_last_annual_physical_time,
breast_cancer_screening: payload.breast_cancer_screening,
breast_cancer_screening_date: payload.breast_cancer_screening_date,
well_child_visit_3_18: payload.well_child_visit_3_18,
well_child_visit_3_18_date: payload.well_child_visit_3_18_date,
well_child_first_30: payload.well_child_first_30,
well_child_first_30_date: payload.well_child_first_30_date,
seven_day_hospital_follow_up: payload.seven_day_hospital_follow_up,
seven_day_hospital_follow_up_date:
payload.seven_day_hospital_follow_up_date,
thirty_day_hospital_follow_up: payload.thirty_day_hospital_follow_up,
thirty_day_hospital_follow_up_date:
payload.thirty_day_hospital_follow_up_date,
emergency_dept_follow_up: payload.emergency_dept_follow_up,
emergency_dept_follow_up_date: payload.emergency_dept_follow_up_date,
diabetes_screening_schizo_bipolar:
payload.diabetes_screening_schizo_bipolar,
diabetes_screening_schizo_bipolar_date:
payload.diabetes_screening_schizo_bipolar_date,
potential_avoidable_ed_utilization:
payload.potential_avoidable_ed_utilization,
potential_avoidable_ed_utilization_date:
payload.potential_avoidable_ed_utilization_date,
cervical_cancer_screening: payload.cervical_cancer_screening,
cervical_cancer_screening_date: payload.cervical_cancer_screening_date,
csc_eye: payload.csc_eye,
csc_eye_date: payload.csc_eye_date,
amr: payload.amr,
amr_date: payload.amr_date,
col: payload.col,
col_date: payload.col_date,
},
outpatient_mental_health: {
contact_needs: payload.outpatient_mental_contact_needs,
contact_type: payload.outpatient_mental_contact_type,
number_of_contacts: payload.outpatient_mental_number_of_contacts,
appointment_date: payload.outpatient_mental_appointment_date,
appointment_time: payload.outpatient_mental_appointment_time,
contact_date: payload.outpatient_mental_contact_date,
contact_time: payload.outpatient_mental_contact_time,
contact_details: payload.outpatient_mental_contact_details,
contact_duration: payload.outpatient_mental_contact_duration,
number_of_contacts: payload.outpatient_mental_number_of_contacts,
type_of_appointment: payload.outpatient_mental_type_of_appointment,
diagnosis: payload.outpatient_mental_diagnosis,
grpa: payload.outpatient_mental_grpa,
grpa_date_completed: payload.outpatient_mental_grpa_date_completed,
grpa_frequency: payload.outpatient_mental_grpa_frequency,
noms: payload.outpatient_mental_noms,
noms_date_completed: payload.outpatient_mental_noms_date_completed,
noms_frequency: payload.outpatient_mental_noms_frequency,
},
outpatient_substance_use: {
contact_needs: payload.outpatient_substance_contact_needs,
contact_type: payload.outpatient_substance_contact_type,
number_of_contacts: payload.outpatient_substance_number_of_contacts,
appointment_date: payload.outpatient_substance_appointment_date,
appointment_time: payload.outpatient_substance_appointment_time,
contact_date: payload.outpatient_substance_contact_date,
contact_time: payload.outpatient_substance_contact_time,
contact_details: payload.outpatient_substance_contact_details,
contact_duration: payload.outpatient_substance_contact_duration,
number_of_contacts: payload.outpatient_substance_number_of_contacts,
type_of_appointment: payload.outpatient_substance_type_of_appointment,
diagnosis: payload.outpatient_substance_diagnosis,
grpa: payload.outpatient_substance_grpa,
grpa_date_completed: payload.outpatient_substance_grpa_date_completed,
grpa_frequency: payload.outpatient_mental_grpa_frequency,
noms: payload.outpatient_substance_noms,
noms_date_completed: payload.outpatient_substance_noms_date_completed,
noms_frequency: payload.outpatient_mental_noms_frequency,
},
})
if (!payload.duplicate_skip) {
// Check if there is already a referral with same first name, last name and dob. If so return an error
await Referral.find(
{
$and: [
{
last_name: {
$regex: new RegExp(`^${payload.last_name}$`, "i"),
},
first_name: {
$regex: new RegExp(`^${payload.first_name}$`, "i"),
},
dob: {
$regex: new RegExp(`^${payload.dob}$`, "i"),
},
project: {
$regex: new RegExp(`^${payload.project}$`, "i"),
},
},
],
},
(err, response) => {
if (response.length !== 0) {
var message = {
message:
"Referral with the same First Name, Last Name and Date of Birth already exists.",
type: "duplicate",
}
duplicate = true
return res.status(400).json(message)
}
},
)
} else {
new_referral
.save()
.then((referral) => {
// If an agency adds a referral with 'Agency' field blank / None, send CBHS email that new referral was added
if (
referral.referral_agency === "None" ||
referral.referral_agency === "CBHS" ||
referral.referral_agency === ""
) {
const send_to = "CBHS"
updateAgencyNotification(send_to, referral.project[0])
}
return res.status(200).json(referral)
})
.catch((err) => {
// Validation error, missing keys on required fields for referral creation
logger.error(req.originalUrl + err)
if (err.name === "ValidationError") {
missing_keys = []
for (err in err.errors) {
switch (err) {
case "referral_source":
missing_keys.push("Referral Source")
break
case "referral_source_email":
missing_keys.push("Referral Source Email")
break
case "referral_need":
missing_keys.push("Referral Need")
break
case "last_name":
missing_keys.push("Last Name")
break
case "first_name":
missing_keys.push("First Name")
break
case "middle_name":
missing_keys.push("Middle Name")
break
case "gender":
missing_keys.push("Gender")
break
case "address_1":
missing_keys.push("Address 1")
break
case "county":
missing_keys.push("County")
break
case "zip":
missing_keys.push("Zipcode")
break
case "phone":
missing_keys.push("Phone Number")
break
}
}
const validationError = {
message: "Validation Error",
fields: missing_keys,
}
return res.status(400).json(validationError)
}
})
}
})
problem here was there was no logic to save when duplicate skip was false.
if (!payload.duplicate_skip)
so it looked for something with the same first, last and dob,
await Referral.find(
{
$and: [
{
last_name: {
$regex: new RegExp(`^${payload.last_name}$`, "i"),
},
first_name: {
$regex: new RegExp(`^${payload.first_name}$`, "i"),
},
dob: {
$regex: new RegExp(`^${payload.dob}$`, "i"),
},
project: {
$regex: new RegExp(`^${payload.project}$`, "i"),
},
},
],
}
and if there was no match, it just did nothing.
the logic to save
else {
new_referral
.save()
.then((referral) => {
// If an agency adds a referral with 'Agency' field blank / None, send CBHS email that new referral was added
if (
referral.referral_agency === "None" ||
referral.referral_agency === "CBHS" ||
referral.referral_agency === ""
) {
const send_to = "CBHS"
updateAgencyNotification(send_to, referral.project[0])
}
return res.status(200).json(referral)
})
.catch((err) => {
// Validation error, missing keys on required fields for referral creation
logger.error(req.originalUrl + err)
if (err.name === "ValidationError") {
missing_keys = []
for (err in err.errors) {
switch (err) {
case "referral_source":
missing_keys.push("Referral Source")
break
case "referral_source_email":
missing_keys.push("Referral Source Email")
break
case "referral_need":
missing_keys.push("Referral Need")
break
case "last_name":
missing_keys.push("Last Name")
break
case "first_name":
missing_keys.push("First Name")
break
case "middle_name":
missing_keys.push("Middle Name")
break
case "gender":
missing_keys.push("Gender")
break
case "address_1":
missing_keys.push("Address 1")
break
case "county":
missing_keys.push("County")
break
case "zip":
missing_keys.push("Zipcode")
break
case "phone":
missing_keys.push("Phone Number")
break
}
}
const validationError = {
message: "Validation Error",
fields: missing_keys,
}
return res.status(400).json(validationError)
}
})
}
was never triggered when skip was false
so i added it after
if (response.length !== 0) {
var message = {
message:
"Referral with the same First Name, Last Name and Date of Birth already exists.",
type: "duplicate",
}
duplicate = true
return res.status(400).json(message)
}
meaning, "after checking for a duplicate (skip is false), if you don't get anything back (response === 0), then go ahead and actually create and save a new instance of the model"

Meteor Mongo Collections find forEach cursor iteration and saving to ElasticSearch Problem

i have Meteor App which is connected to MongoDB.
In mongo i have a table which has ~700k records.
I have a cron job each week, where i read all the records from the table (using Mongo Cursor) and in batches of 10k i want to insert them inside Elastic Search so they are indexed.
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Since for each is synchronous, goes over each record before it continues, and client.bulk is async, this is overloading the elastic search server and it crashes with Out of Memory Exception.
Is there a way to pause the forEach during the time when the insert is being done? I tried async/await but this does not seem to work as well.
let articles = []
Collections.Articles.find({}).forEach(async function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
await client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Any way how to achieve this?
EDIT: I am trying to achieve something like this - if i use promises
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
// Pause FETCHING rows with forEach
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles }).then(() => {
console.log('inserted')
// RESUME FETCHING rows with forEach
console.log("RESUME READING");
})
data = []
}
})
Managed to get this working with ES2018 Async iteration
Got an idea from
Using async/await with a forEach loop
Here is the code that is working
let articles = []
let cursor = Collections.Articles.find({})
for await (doc of cursor) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (articles.length === 10000) {
await client.bulk({ maxRetries: 5, index: 'trusted', type: 'artikel', body: articles })
articles = []
}
}
This works correctly and it manages to insert all the records into Elastic Search without crashing.
If you are concerned with the unthrottled iteration, then may use the internal Meteor._sleepForMs method, that allows you to put a async timeout in your sync-styled code:
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
Now this works fine within the Meteor environment (Meteor.startup, Meteor.methods, Meteor.publish).
You cron is likely to be not within this environment (= Fiber) so you may write a wrapper that binds the environment:
const bound = fct => Meteor.bindEnvironment(fct)
const iterateSlow = bound(function (timeout) {
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
iterateSlow(50) // iterates with 50ms timeout
Here is a complete minimal example, that you can reproduce with a fresh project:
// create a minimal collection
const MyDocs = new Mongo.Collection('myDocs')
// fill the collection
Meteor.startup(() => {
for (let i = 0; i < 100; i++) {
MyDocs.insert({})
}
})
// bind helper
const bound = fct => Meteor.bindEnvironment(fct)
// iterate docs with interval between
const iterateSlow = bound(function (timeout) {
MyDocs.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
// simulate external environment, like when cron runs
setTimeout(() => {
iterateSlow(50)
}, 2000)

Where to find Mongo Collection in Express?

I'm reviewing some old code I wrote in Express/Mongo/Mongoose (based on an online tutorial) and can't locate the portion of code which dictates which Collection in MongoDB gets written to.
I have a Database, UsersDB and within that database there are several Collections. The Collection that keeps growing every time the code executes user.save is the users Collection. But I can't find any reference in my code base to the users collection.
The only place in my code where I save a User is:
var app = require('../app');
var util = require('util');
var User = require('../models/user'),
Auth = User.Auth,
Name = User.Name,
Email= User.Email,
Phone = User.Phone,
Address = User.Address,
Company = User.Company,
PersonalData = User.PersonalData,
Id = User.Id,
Photo = User.Photo,
Member = User.Member,
CreditCard = User.CreditCard,
UserObj = User.User;
var moment = require('moment');
var async = require('async');
. . .
. . .
exports.user_create_post = [
(req,res, next) => {
console.log("Request: " + util.inspect(req.body));
},
//VALIDATE
body('mainEmail', 'Must be valid email.').isLength({min: 5}).trim(),
//SANITIZE
sanitizeBody('*').escape(),
//POPULATE NEW DOCUMENT
(req,res,next) => {
const errors = validationResult(req);
var auth = new Auth(
dateEffective: {value: moment(Date.now()).format("YYYY-MM-DD hh:mm:ss SSS"), attr: {hidden: true, label: ""}},
username: {"value": req.body.username, "attr": {hidden: false, label: "Username: "}},
password: {"value": req.body.password, "attr": {hidden: false, label: "Password: "}},
mainEmail: {"value": req.body.mainEmail, "attr": {hidden: false, label: "Email: "}}
});
var user = new UserObj(
{authData: [auth]}
);
if (!errors.isEmpty()) {
const errorFormatter = ({ location, msg, param, value, nestedErrors }) => {
// Build your resulting errors however you want! String, object, whatever - it works!
return `${location}[${param}]: ${msg}`;
};
const result = validationResult(req).formatWith(errorFormatter);
if (!result.isEmpty()) {
return res.json({ errors: result.array() });
}
}
else {
user.save(function(err){
if (err) { return next(err);}
});
res.redirect("http://localhost:4200/two-fa/"+user._id);
}
}
I also have a Models module (user.js):
. . .
. . .
. . .
module.exports = {
Auth: mongoose.model('Auth', AuthSchema),
Name: mongoose.model('Name', NameSchema),
Email: mongoose.model('Email', EmailSchema),
Phone: mongoose.model('Phone', PhoneSchema),
Address: mongoose.model('Address', AddressSchema),
Company: mongoose.model('Company', CompanySchema),
PersonalData: mongoose.model('PersonalData', PersonalDataSchema),
Id: mongoose.model('Id', IdSchema),
Photo: mongoose.model('Photo', PhoneSchema),
Member: mongoose.model('Member', MemberSchema),
CreditCard: mongoose.model('CreditCard', CreditCardSchema),
User: mongoose.model('User', UserSchema)
}
I did a search on my entire code, and nowhere is there any mention of users, which is the Collection that's getting written to.
Where should I look to try to trace how the users collection is getting written to?
Thank you!
As per how-to-access-a-preexisting-collection-with-mongoose, if a 3rd argument to mongoose.model is not provided, then Mongo automatically "pluralizes" the model name to give the name of the Collection.

open REST cypher transaction

Trying to write a script to open a transaction in groovy. Currently have:
def static transaction(statement, params,success, error)
{
def http = new HTTPBuilder( 'http://localhost:7474' )
http.request( POST, JSON ) {
uri.path = '/db/data/transaction'
headers.'X-Stream' = 'true'
requestContentType = JSON
body = [ statements : statement , params : params ?: [:] ]
// uri.query = [ param : 'value' ]
response.success = { resp, json ->
if (success) success(json)
else {
println "Status ${resp.statusLine} Columns ${json.columns}\nData: ${json.data}"
}
}
response.failure = { resp, message ->
def result=[status:resp.statusLine.statusCode,statusText:resp.statusLine.reasonPhrase]
result.headers = resp.headers.collect { h -> [ (h.name) : h.value ] }
result.message = message
if (error) {
error(result)
} else {
println "Status: ${result.status} : ${result.statusText} "
println 'Headers: ${result.headers}'
println 'Message: ${result.message}'
}
}
}
}
transaction("start n=node(*) return n",[id:56981],{ println "Success: ${it}" },{ println "Error: ${it}" })
However I am getting the following error in the response:
Success: [commit:http://localhost:7474/db/data/transaction/4/commit, results:[], errors:[[code:40001, status:INVALID_REQUEST_FORMAT, message:Unable to deserialize request. Expected [START_OBJECT, FIELD_NAME, START_ARRAY], found [START_OBJECT, FIELD_NAME, VALUE_STRING].]]]
Thoughts? Thanks!
Please check out https://gist.github.com/7053223. Basically you did not build the json structure correctly. According to http://docs.neo4j.org/chunked/milestone/rest-api-transactional.html a json map containing statements key is sent. Inside there's an array of maps containing a statement and parameters key. Compare l.15 vs l.16-18 to see the difference.
Faced same problem with php, statements must contain array of statement:
[
statements => [
statemant
],
resultDataContents => [
'row',
'graph',
],
'includeStats' => true,
]