I would like to find a solution to my problem: I have a subscription from external service that send a lot of information that I need to save on mongoDB and get the stored in a dashboard. Before to insert them I need to find if exist and update instead create.
This is my code:
await ProjectModel.findOne( { address } ).then( project => {
if( project ) {
project.tasked?.push( task )
const index = ( project.owners as TypeOwners ).findIndex( ( p ) => p.owner === task.owner )
if( index >= 0 && project.owners ) {
project.owners[index].amount++
project.owners[index].tasks.push( task.address )
project.markModified('owners');
} else {
project.owners?.push( {
tasks: [ task.address ],
amount: 1,
owner: task?.owner
} )
}
project?.save()
console.log( `updated: ${address}` )
} else {
ProjectModel.create( {
cm_address: cm_address,
platform: platform,
project_symbol: symbol,
project_collection:name,
creator: creators as Creator[],
description: description,
tasked: [ tasked ],
owners: [ {
tasks: [ task.address ],
amount: 1,
owner: task?.owner
}]
})
console.log( `created: ${address}` )
}
})
The problem is some time the information received are simultaneous and don't give the time to mongodb to check if exist and duplicate the record (instead of update) and some time I receive this error:
VersionError: No matching document found for id "XXXXXXXX" version 0 modifiedPaths ....
Someone have a solution for this situation? Thank you :slight_smile:
Related
Trying to correctly troubleshoot HTTP status codes and responses on upsert. I've scoured SO and google to find examples of this behavior, but the answers are always really old (pre Sequelize 6 era).
From what I gather over at Sequelize docs, UPSERT returns Promise<Model,boolean | null> and additionally For SQLite/Postgres, created value will always be 'null'. But apparently for me, whether it created/didn't update/updated, it always returns null
Am I correctly understanding that boolean return will return for update(T)/no update(F) and null for creation? Or will it only return null whether create|update|no update for postgres users?
This is MY expected return behavior:
data: [
{
id: 64920,
...etc...
},
null // null = created
],
data: [
{
id: 64921,
...etc...
},
false // false = no update (data is the same)
],
data: [
{
id: 64922,
...etc...
},
true // true = update complete (some data was new)
]
Sample controller upsert command:
exports.createOne = async (req, res, next) => {
Trip.upsert({
id: req.body.id,
...etc...
},{returning: true } // not needed, default in Sequelize =< 6.0
)
.then(function (test) {
if (test) {
res.status(200);
res.send("Successfully stored");
} else {
res.status(200);
res.send("Successfully inserted");
}
})
.catch((err) => {
res.status(500).json({ error: err.message });
});
};
Anyone have examples or suggestions to properly implement this?
Thank you!
i got this error : Encountered two children with the same key, 610bebf8b6f3820b38b0c613. Keys should be unique so that components maintain their identity across updates. Non-unique keys may cause children to be duplicated and/or omitted — the behavior is unsupported and could change in a future version
but i use mongodb id so no way 2 id be them same
it happen when i load more items...
im using redux toolkit express mongoose
it happen sometimes not always
flatList :
<View style={{flex: 1, margin: 10, flexDirection: 'column'}}>
<Search />
{users.pending && !users.users.length ? (
<ActivityIndicator size="large" color="#0000ff" />
) : users.users.length === 0 ? (
<Text>No users</Text>
) : (
<FlatList
showsVerticalScrollIndicator={false}
showsHorizontalScrollIndicator={false}
data={users.users}
renderItem={({item}) => <Card user={item} />}
keyExtractor={item => item._id}
onEndReachedThreshold={0}
onEndReached={handleLoadMore}
/>
)}
</View>
usersSlice :
getUsersSuccess: (state, action) => {
state.pending = false;
if (state.users.length === 0) {
state.users = action.payload;
} else {
state.users = [...state.users, ...action.payload];
}
state.skip = state.users.length;
},
apiCall
export const getUsers = async (dispatch, skip) => {
dispatch(getUsersStart());
try {
const res = await axios.get(`http://10.0.2.2:5000/users?skip=${skip}`);
dispatch(getUsersSuccess(res.data));
} catch (error) {
dispatch(getUsersError());
console.error('error => ', error.message);
}
};
server :
export const getUsers = async (req, res) => {
const skip =
req.query.skip && /^\d+$/.test(req.query.skip) ? Number(req.query.skip) : 0;
console.log(' req.qury.skip >>>>', req.query.skip);
try {
const users = await User.find({}, undefined, { skip, limit: 7 });
users.map((user) => {
user.name = decryptString(user.name, process.env.CRYPTO_PASSWORD);
});
res.status(200).json(users);
} catch (error) {
console.error(error.message);
res.status(500).json('Server Error');
}
};
i get
virtualizedList: You have a large list that is slow to update - make sure your renderItem function renders components that follow React performance best practices like PureComponent, shouldComponentUpdate, etc.
i think about delete duplicate item but i think not good choice
but i use mongodb id so no way 2 id be them same it happen when i load more items
your assumption can be wrong in multiple ways. Mongo IDs are unique in the database, but you could be getting duplicate values because of a variety of reasons.
I suspect that the API is giving you overlapping data and you merge new data with the old data without checking -
state.users = [...state.users, ...action.payload];
modify this to have a basic uniqueness check - and only add elements that are not already in the old list
i have Meteor App which is connected to MongoDB.
In mongo i have a table which has ~700k records.
I have a cron job each week, where i read all the records from the table (using Mongo Cursor) and in batches of 10k i want to insert them inside Elastic Search so they are indexed.
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Since for each is synchronous, goes over each record before it continues, and client.bulk is async, this is overloading the elastic search server and it crashes with Out of Memory Exception.
Is there a way to pause the forEach during the time when the insert is being done? I tried async/await but this does not seem to work as well.
let articles = []
Collections.Articles.find({}).forEach(async function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
await client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles })
data = []
}
})
Any way how to achieve this?
EDIT: I am trying to achieve something like this - if i use promises
let articles = []
Collections.Articles.find({}).forEach(function(doc) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (0 === articles.length % 10000) {
// Pause FETCHING rows with forEach
client.bulk({ maxRetries: 5, index: 'main', type: 'article', body: articles }).then(() => {
console.log('inserted')
// RESUME FETCHING rows with forEach
console.log("RESUME READING");
})
data = []
}
})
Managed to get this working with ES2018 Async iteration
Got an idea from
Using async/await with a forEach loop
Here is the code that is working
let articles = []
let cursor = Collections.Articles.find({})
for await (doc of cursor) {
articles.push({
index: {_index: 'main', _type: 'article', _id: doc.id }
},
doc);
if (articles.length === 10000) {
await client.bulk({ maxRetries: 5, index: 'trusted', type: 'artikel', body: articles })
articles = []
}
}
This works correctly and it manages to insert all the records into Elastic Search without crashing.
If you are concerned with the unthrottled iteration, then may use the internal Meteor._sleepForMs method, that allows you to put a async timeout in your sync-styled code:
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
Now this works fine within the Meteor environment (Meteor.startup, Meteor.methods, Meteor.publish).
You cron is likely to be not within this environment (= Fiber) so you may write a wrapper that binds the environment:
const bound = fct => Meteor.bindEnvironment(fct)
const iterateSlow = bound(function (timeout) {
Collections.Articles.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
iterateSlow(50) // iterates with 50ms timeout
Here is a complete minimal example, that you can reproduce with a fresh project:
// create a minimal collection
const MyDocs = new Mongo.Collection('myDocs')
// fill the collection
Meteor.startup(() => {
for (let i = 0; i < 100; i++) {
MyDocs.insert({})
}
})
// bind helper
const bound = fct => Meteor.bindEnvironment(fct)
// iterate docs with interval between
const iterateSlow = bound(function (timeout) {
MyDocs.find().forEach((doc, index) => {
console.log(index, doc._id)
Meteor._sleepForMs(timeout)
})
return true
})
// simulate external environment, like when cron runs
setTimeout(() => {
iterateSlow(50)
}, 2000)
I am trying to create a meteor RESTful API for my app based on this The Meteor Chef online tutorial. The HTTP package is installed in the beginning of the tutorial, in order to test the RESTful API once the API development is completed.
I am currently in the testing phase and cant seem to get my GET Methods used to retrieve data from my collection to work.
Find below my GET Method code:
methods: {
pescrow: {
GET: function( context, connection ) {
var hasQuery = API.utility.hasData( connection.data );
console.log("hasQuery value == " +hasQuery+ " on line 183");
if ( hasQuery ) {
connection.data.owner = connection.owner;
console.log("Your in GET::hasQuery: Line 187 " + connection.data );
var getPescrows = recipientsDetails.find( connection.data ).fetch();
console.log("getPescrows value: " +getPescrows+ " Line 203");
if ( getPescrows.length > 0 ) {
// We found some pescrows, we can pass a 200 (success) and return the
// found pescrows.
console.log("getPescrows found Line 205");
API.utility.response( context, 200, getPescrows );
}
else {
console.log("getPescrows NOT found Line 208!");
// Bummer, we didn't find any pescrows. We can pass a 404 (not found)
// and return an error message.
API.utility.response( context, 404, { error: 404, message: "No Pescrows found, dude." } );
}
}
else {
// Our request didn't contain any params, so we'll just return all of
// the pescrows we have for the owner associated with the passed API key.
var getPescrows = recipientsDetails.find( { "owner": connection.owner } ).fetch();
API.utility.response( context, 200, getPescrows );
}
}
}
}
I test my API via the Chrome console by pasting in the below code:
HTTP.get( "http://localhost:8000/paymentC2B/v1", {
params: {
"api_key": "b21d83ef267bd829a9d732551270c718",
"paymentStatus": "Pending",
"recipientNumber" : "0705087633"
}
}, function( error, response ) {
if ( error ) {
console.log( error );
} else {
console.log( response );
}
});
And the response I get in the terminal is:
hasQuery == true Line 183
Your in GET::hasQuery: Line 187 [object Object]
getPescrows value: Line 203
getPescrows NOT found Line 208!
When I run the query below in the console it successfully yields:
recipientsDetails.find({paymentStatus:"Pending", recipientNumber: "0705087633"}, {sort: {paymentDate: 'desc' }}).fetch()
Showing:
[{…}]
0
:
key : "b21d83ef267bd829a9d732551270c718"
paymentDate : "2018-04-02 15:15:49"
paymentStatus : "Pending"
recipientAmount : "500"
recipientNumber : "0705087633"
_id : "uSsCbdBmmhR2AF2cy"
__proto__ : Object
length : 1
__proto__ : Array(0)
It seems like the issue is in the recipientsDetails.find( connection.data ).fetch(); query. Can someone kindly point out where I am going wrong in my code?
Looking forward to your response.
When you test your params include api_key. I'm betting this key does not appear in your recipientsDetails collection.
Instead of just doing:
connection.data.owner = connection.owner;
Try:
connection.data.owner = connection.owner;
delete connection.data.api_key;
Here's what i did
static populateReferralLinks(){
return Promise.coroutine(function*(){
let companies = yield Company.find({},'billing referral current_referral_program')
.populate('billing.user','emails name');
for(let i = 0 ; i < length ; i++){
companies[i].referral.is_created = true;
companies[i].referral.referral_email = companies[i].billing.user.emails[0].email;
companies[i] = yield companies[i].save();
}
return companies;
}).apply(this)
.catch((err) => {
throw err;
});
}
I have a funciton in which i am selecting only 3 fields to go ahead with i.e billing,current_referral_program and referral.
And populating user using the reference stored in billing.user.
Now when i call this function then on line
companies[i].save();
The following command is shown in the terminal in windows
Mongoose: companies.update(
{ _id: ObjectId("58d12e1a588a96311075c45c") },
{ '$set':
{ billing:
{ configured: false,
user: ObjectId("58d12e16588a96311075c45a") },
referral:
{ is_created: true,
referral_email: 'jadon.devesh98#gmail.com',
},
updatedAt: new Date("Wed, 22 Mar 2017 12:02:55 GMT")
}
}
)
But in Mac's terminal it shows this command
Mongoose: companies.update({ _id: ObjectId("58d12e1a588a96311075c45c") }) { '$set': { billing: { configured: false, user: ObjectId("58d12e16588a96311075c45a") }, current_limit: {}, current_usage: {},referral: { is_created: true, referral_email: 'jadon.devesh98#gmail.com'}}, '$unset': { updatedAt: 1 } }
Now, I haven't mentioned current_limit and current_usage to be empty. it's executing fine on windows but on Mac it's setting current_limit and current_usage empty thus updating my document with empty objects on Mac but not on windows.
It should behave same way on both OS but it is not.
Apparently this problem was there in Mongoose 4.5.8 and is resolved in the latest version i.e 4.9.1
Check it here