High frequency calls leads to duplicates with findOrCreate in Waterline & Sails - postgresql

How to handle high frequency updateOrCreate requests with Waterline in Sails for a Postgresql database ?
I tried to use findOrCreate and then update the item, I tried findOne and then update or create the item, I tried to put a beforeCreate, a beforeValidation hook method to check if the item exists but without any success.
Should I add an error handler to get errors from the unique index and try again?
In the Waterline docs, there is a warning about it but no direction to solve this problem.
Thank you for any tips.

Should I add an error handler to get errors from the unique index and try again?
That's going to be the only option until such time as Waterline implements transactions. Something like:
// This will hold the found or created user
var user;
// Keep repeating until we find or create a user, or get an error we dont expect
async.doUntil(
function findOrCreate(cb) {
// Try findOrCreate
User.findOrCreate(criteria, values).exec(function(err, _user) {
// If we get an error that is not a uniqueness error on the
// attribute we expect collisions on, bail out of the doUntil
if (err &&
(
!err.invalidAttributes["myUniqueAttribute"] ||
!_.find(err.invalidAttributes["myUniqueAttribute"], {rule: 'unique'})
)
) {
return cb(err);
}
// Otherwise set the user var
// It may still be undefined if a uniqueness error occurred;
// this will just cause doUntil to run this function again
else {
user = _user;
return cb();
}
},
// If we have a user, we are done. Otherwise go again.
function test() {return user},
// We are done!
function done(err) {
if (err) {return res.serverError(err);}
// "user" now contains the found or created user
}
});
Not the prettiest, but it should do the trick.

Related

How mongoose middleware works and what is next()?

userSchema.pre('save',async function(next){
//hash the password before saving user to database
next()
})
Hey guys I'm trying to understand the concept of middleware in mongoose. Assuming that I have an userSchema that I run the pre hook on to hash the password before saving the user to the database. On the surface, as far as I understand, the above code that I have will hash the password (not the important part for this question) and then call next() to signal that the function is done. However, I am trying to understand how things work under the hood. I want to know what is next() ? Can you guys walk me through an example of how everything works together under the hood from start to end once the code get executed or somehow help me to have a better understanding of this concept? Thanks
Short : with the pre method you can register listeners for certain events of your Schemas. So pre('save', callback) will fire whenever you save a document of said Model. pre means it will execute before the event, so it can be used (for example) to hash a password before saving it to the document.
However, you have several options to define them, see below :
The combination of using an async callback function and providing the next parameter is not necessary, you can either :
use normal callback with next parameter
the next parameter is a function provided to you by mongoose to have a way out, or to tell mongoose you are done and to continue with the next step in the execution chain. Also it is possible to pass an Error to next it will stop the execution chain.
schema.pre('save', function(next) {
// do stuff
if (error) { return next(new Error("something went wrong"); }
return next(null);
});
use async callback
Here the execution chain will continue once your async callback has finished. If there is an error and you want to break/stop execution chain you just throw it
schema.pre('save', async function() {
// do stuff
await doStuff()
await doMoreStuff()
if (error) { throw new Error("something went wrong"); }
return;
});
Straight from the docs : https://mongoosejs.com/docs/middleware.html#pre
Example
const { Schema, model } = require('mongoose');
const SomeSchema = new Schema ({
name : { type : String }
});
SomeSchema.pre('save', function (next) {
console.log('pre save fired!');
return next();
});
const Some = model('Some', SomeSchema);
console.log('before create');
const doc = new Some({ name : 'test' });
doc.save((err) => {
console.log('after saved');
});
This will output
before create
pre save fired!
after saved

How to stop the user from entering the duplicate record on default save

I have a custom module where there is an email field. Now i want to stop the user if the email is already in the database.
I want to stop the user on save button and show the error. Like when a required field goes empty.
I tried to get some help but was not able to understand it.
Note: I realized after posting this that you are using suitecrm which this answer will not be applicable toward but I will leave it in case anyone using Sugar has this question.
There are a couple of ways to accomplish this so I'll do my best to walk through them in the order I would recommend. This would apply if you are using a version of Sugar post 7.0.0.
1) The first route is to manually create an email address relationship. This approach would use the out of box features which will ensure your system only keeps track of a single email address. If that would work for your needs, you can review this cookbook article and let me know if you have any questions:
https://support.sugarcrm.com/Documentation/Sugar_Developer/Sugar_Developer_Guide_9.2/Cookbook/Adding_the_Email_Field_to_a_Bean/
2) The second approach, where you are using a custom field, is to use field validation. Documentation on field validation can be found here:
https://support.sugarcrm.com/Documentation/Sugar_Developer/Sugar_Developer_Guide_9.2/Cookbook/Adding_Field_Validation_to_the_Record_View/index.html
The code example I would focus on is:
https://support.sugarcrm.com/Documentation/Sugar_Developer/Sugar_Developer_Guide_9.2/Cookbook/Adding_Field_Validation_to_the_Record_View/#Method_1_Extending_the_RecordView_and_CreateView_Controllers
For your example, I would imagine you would do something like this:
Create a language key for your error message:
./custom/Extension/application/Ext/Language/en_us.error_email_exists_message.php
<?php
$app_strings['ERROR_EMAIL_EXISTS_MESSAGE'] = 'This email already exists.';
Create a custom controller for the record creation (you may also want to do this in your record.js):
./custom/modules//clients/base/views/create/create.js
({
extendsFrom: 'RecordView',
initialize: function (options) {
this._super('initialize', [options]);
//reference your language key here
app.error.errorName2Keys['email_exists'] = 'ERROR_EMAIL_EXISTS_MESSAGE';
//add validation tasks
this.model.addValidationTask('check_email', _.bind(this._doValidateEmail, this));
},
_doValidateEmail: function(fields, errors, callback) {
var emailAddress = this.model.get('your_email_field');
//this may take some time so lets give the user an alert message
app.alert.show('email-check', {
level: 'process',
title: 'Checking for existing email address...'
});
//make an api call to a custom (or stock) endpoint of your choosing to see if the email exists
app.api.call('read', app.api.buildURL("your_custom_endpoint/"+emailAddress), {}, {
success: _.bind(function (response) {
//dismiss the alert
app.alert.dismiss('email-check');
//analyze your response here
if (response == '<email exists>') {
errors['your_email_field'] = errors['your_email_field'] || {};
errors['your_email_field'].email_exists = true;
}
callback(null, fields, errors);
}, this),
error: _.bind(function (response) {
//dismiss the alert
app.alert.dismiss('email-check');
//throw an error alert
app.alert.show('email-check-error', {
level: 'error',
messages: "There was an error!",
autoClose: false
});
callback(null, fields, errors);
})
});
},
})
Obviously, this isn't a fully working example but it should get you most of the way there. Hope this helps!

Azure Mobile Offline Sync: Cannot delete an operation from __operations

I'm having a huge issue that I've been trying for days to get through. I have a scenario in which I'm trying to handle an Insert Conflict in my Xamarin project. The issue is that the record in the Cloud DB doesn't exist because there was an issue with a foreign key constraint so I'm in a scenario in which the sync conflict handler needs to delete the local record along with the record in the __operations table in SQLite. I've tried everything. Purge with the override set to 'true' so that it should delete the local record and all operations associated. Doesn't work. I've been just trying to force delete it by accessing the SQL store manually:
var id = localItem[MobileServiceSystemColumns.Id];
var operationQuery = await store.ExecuteQueryAsync("__operations", $"SELECT * FROM __operations WHERE itemId = '{id}'", null).ConfigureAwait(false);
var syncOperation = operationQuery.FirstOrDefault();
var tableName = operation.Table.TableName;
await store.DeleteAsync(tableName, new List<string>(){ id.ToString() });
if (syncOperation != null)
{
await store.DeleteAsync("__operations", new List<string>() { syncOperation["id"].ToString() }).ConfigureAwait(false);
}
I am able to query the __operations table and I can see the ID of the item I want to delete. The DeleteAsync method runs without exception but no status is returned so I have no idea if this worked or not. When I try to sync again the operation stubbornly exists. This seems ridiculous. How do I just delete an operation without having to sync with the web service? I'm about to dig down further and try to force it even harder by using the SQLiteRaw library but I'm really really hoping I'm missing something obvious? Can anyone help? THANKS!
You need to have a subclass of the Microsoft.WindowsAzure.MobileServices.Sync.MobileServiceSyncHandler class, which overrides OnPushCompleteAsync() in order to handle conflicts and other errors. Let's call the class SyncHandler:
public class SyncHandler : MobileServiceSyncHandler
{
public override async Task OnPushCompleteAsync(MobileServicePushCompletionResult result)
{
foreach (var error in result.Errors)
{
await ResolveConflictAsync(error);
}
await base.OnPushCompleteAsync(result);
}
private static async Task ResolveConflictAsync(MobileServiceTableOperationError error)
{
Debug.WriteLine($"Resolve Conflict for Item: {error.Item} vs serverItem: {error.Result}");
var serverItem = error.Result;
var localItem = error.Item;
if (Equals(serverItem, localItem))
{
// Items are the same, so ignore the conflict
await error.CancelAndUpdateItemAsync(serverItem);
}
else // check server item and local item or the error for criteria you care about
{
// Cancels the table operation and discards the local instance of the item.
await error.CancelAndDiscardItemAsync();
}
}
}
Include an instance of this SyncHandler() when you initialize your MobileServiceClient:
await MobileServiceClient.SyncContext.InitializeAsync(store, new SyncHandler()).ConfigureAwait(false);
Read up on the MobileServiceTableOperationError to see other conflicts you can handle as well as its methods to allow resolving them.

Meteor - no more callbacks for "findOne" function

i'm working on a Meteor project, and I must say that isn't easy at all, especially for one thing: callbacks !
Everything is async, so I wonder how do I must do to get results from my mongodb.
var user = Meteor.users.findOne({username: "john"});
return (user); // sometimes returns "undefined"
...
var user = Meteor.users.findOne({username: "john"});
if (user) // so ok, I check if it exists!
return (user); // Cool, I got my user!
return (); // Ok and what should I return here? I want my user!
I don't want to be dirty and put like setTimeout everywhere.
Anybody has a solution for this ?
EDIT :
I noticed in router.js with console.log that my data is returned 4 times. 2 times with an undefined value and 2 other times with the expected value. In the view, it's still undefined.
Why the router passes like 4 times in this route ? Does it display the first result of the return value in the router ?
What should I return if the find() doesn't find anything ?
EDIT 2: Here is some code to understand.
this.route('profilePage', {
path: 'profil/:_id?',
waitOn: function() {
return [
Meteor.subscribe('article', { prop: this.params._id}), // id can be id or username
Meteor.subscribe('article', { userId: this.params._id}), // id can be id or username
Meteor.subscribe('params'),
Meteor.subscribe('profil', (this.params._id ? this.params._id : Meteor.userId()))
];
},
data: function() {
if (this.params._id) {
var user = Meteor.users.findOne(this.params._id);
if (!user)
user = Meteor.users.findOne({username: this.params._id});
console.log(user);
return user;
}
else if (Meteor.userId())
return Meteor.user();
else
Router.go("userCreate");
}
});
I get this on the console:
http://puu.sh/debdJ/69419911f7.png
(text version following)
undefined
undefined
Object_id: "o3mgLcechYTtHPELh"addresses: (....)
Object_id: "o3mgLcechYTtHPELh"addresses: (....)
findOne(yourId) is a sync method which is equivalent to find({ _id: yourId}, callback). The difference is that find() allows you to define a callback. If you don't pass a callback to find() this method will be sync.
check wrapAsync: http://docs.meteor.com/#/full/meteor_wrapasync
It allows you to code in a sync style with a async operations.
Free lesson on EventedMind: https://www.eventedmind.com/feed/meteor-meteor-wrapasync
My experience thus far is that the Meteor Mongodb package is that the functions do not generally provide callbacks (for some reason insert does...), the functions are atomic (thus sync).
There are meteor packages that can make Mongodb async if you want (I havn't tried any).
I guess this sync approach is in line with the simple maintenance goal of Mongodb. Thinking about it, one of my pet peeves using Node is working with async callback waterfalls/nests, they are a pain to create and maintain... and hopefully this will make my code easier to read and understand and change...
var future = new Future();
var _h = Hunts.findOne({huntId});
if(_h) {
future.return(_h)
} else {
return future.wait();
}
on server/startup.js you need:
Future = Npm.require('fibers/future');

Insert or update record in HighScore table

I have a table that receives high score entries. However, if the user already has an entry in the table (tracked through a GUID field, not the user parameter) I want to update it if the new entry has a better time otherwise don't change the existing record. However, if the user doesn't have a record in the high score table then add a new record. I also have two query parameters to pass to the query.
I want the insert operation to handle this for the table. I have this so far but I get an exception raised when I call InsertAsync(...) on the highscore table
function insert(item, user, request) {
var sql ="select Id from HighScore where PlayerGUID=? AND PlayerBadge=?";
mssql.query(sql, [user.PlayerGUID], [user.PlayerBadge], {
success: function(results) {
if(results.length > 0) {
// leader board record exists so update the current record
// Check the existing record and update it is the new time is better
console.log("Found existing entry");
} else {
// no record exists for this user to insert one
request.execute();
console.log("Found existing entry");
}
}
});
}
Can anyone offer me any assistance with achieving my goal?
Many thanks,
J.
It took some time and some help but here's where I ended up. It works just as I intended it to.
function insert(item, user, request) {
// Store the passed in item object for us when inserting or updating
resultsItem = item;
// Store the request object to allow calld functions to send respond commands
thisRequest = request;
// Retrieve the HighScore table so we can check it for an existing record
hsTable = tables.getTable('HighScore');
// Update the leaderboard
updateLeaderboard(item);
}
// Global variables
var resultsItem, hsTable, thisRequest;
function updateLeaderboard(item){
//Filter the table using the where operator to only include those
// records for the current PlayerGUID and PlayerBadge fields
hsTable.where({
PlayerGUID: item.PlayerGUID,
PlayerBadge: item.PlayerBadge
}).read({
success:updateScore,
error: errorHandler
})
}
function updateScore(results){
if(results.length > 0) {
// If a record already exists then check the PlayerTime
if(results[0].PlayerTime > resultsItem.PlayerTime)
{
// Update the PlayerTime if it is less than the currently saved value
hsTable.update({
id: results[0].id,
PlayerTime: resultsItem.PlayerTime
}, {
success: logSuccess,
error: errorHandler
})
} else {
// Send them OK. Could change this and use the returned code/text to display a custom
// message that tells the user that a previous time is faster.
thisRequest.respond(statusCodes.OK);
}
} else {
// The record for this PlayerGUID and PlayerBadge exists so write one
hsTable.insert({
PlayerName: resultsItem.PlayerName,
PlayerCountry: resultsItem.PlayerCountry,
PlayerTime: resultsItem.PlayerTime,
PlayerBadge: resultsItem.PlayerBadge,
PlayerGender: resultsItem.PlayerGender,
PlayerDOB: resultsItem.PlayerDOB,
PlayerGUID: resultsItem.PlayerGUID
}, {
success: logSuccess,
error: errorHandler
})
}
}
// Called if there is an error
function errorHandler(error){
console.error
("An error occurred trying to update leaderboard infor for player" +
resultsItem.PlayerName);
thisRequest.respond(statusCodes.BAD_REQUEST);
}
//Called if things work out ok.
function logSuccess()
{
thisRequest.respond(statusCodes.OK);
}