I'm building an API using Express and Monk that connects to a database where wrote are mainly handled by a Meteor application.
I know that Meteor uses its own algorithm to generate IDs. So when I do something like that:
id = "aczXLTjzjjn3PchX6" // this is an ID generated by Meteor (not a valid MongoID)
Users.findOne({ _id: id }, function(err, doc) {
console.log(doc);
});
Monk outputs:
Argument passed in must be a single String of 12 bytes or a string of 24 hex characters.
This way, it seems very tricky to me to design a solid and reliable REST API. Thus, I have two questions:
How can I handle the difference in my queries between ids generated by Meteor and valid MongoID()? Is there a simple way to get JSON results from a Meteor database?
Will it be a problem to insert documents from the API which this time will have a valid MongoId()? I will end up with both type of ids in my database, seems very bad to me. :/
As I said in here, in a similar issue you can just override the id converter part of monk:
var idConverter = Users.id; // Keep a reference in case...
Users.id = function (str) { return str; };
But don't expect monk to convert Ids automatically anymore.
How can I handle the difference in my queries between ids generated by Meteor and valid MongoID()? Is there a simple way to get JSON results from a Meteor database?
Nothing much you need to do. When it is a valid ObjectId(mongo db ids) and you got a string just convert it to Object id:
id = ObjectId(id);
User.find(id, ...)
Here's the implementation for monk id method(this.col.id is reference to mongodb native ObjectId):
Collection.prototype.id =
Collection.prototype.oid = function (str) {
if (null == str) return this.col.id();
return 'string' == typeof str ? this.col.id(str) : str;
};
Will it be a problem to insert documents from the API which this time will have a valid MongoId()? I will end up with both type of ids in my database, seems very bad to me. :/
It is bad. Though it won't cause much trouble(in my experience in nodejs) if you be careful. But not all the times you are careful(programmer errors happen a lot), but it's manageable. In static languages(like Java) this is a big NO, because a field can only one type(either string or ObjectId).
My suggestion is that don't use mongodb ObjectId at all and just use strings as ids. On inserts just give it string _id, so the driver won't give it ObjectId. Though you can stop the driver from doing so by overriding pkFactory, but it doesn't seem to be easy with monk.
One more thing is that monk is not actively maintained and it's just a thin layer on top of mongodb. In my experience if you have multiple collections and large/complex code base mongoose will be much better to use.
Just to keep this question updated.
As stated in the docs, Monk is automatically casting the strings into OjbectID.
In order to disable this behaviour without using hacky solutions you'll have to disable that feature. In order to do that you just need to set castIds to false when getting the db.
So:
const Users = db.get('users', { castIds: false });
Now this will work:
Users.findOne({ _id: "aczXLTjzjjn3PchX6" }, function(err, doc) {
console.log(doc);
});
Related
There are a few things going on with Prisma update that I just don't get.
Why is the update (using the ORM way) not performed ?
Why the value of data.address seems to affect the outcome of the update ?
Do I have to provide all of the fields of the entity when updating ? Or in this case, could I just put what I want changed inside of data ?
I am using #prisma/client#3.15.2
Here is what I am currently working with:
const { valid: validFor, expire, state, address, ...safeProperties } = data;
const addressAsUnsigned = address >>> 0; // address is an ip address represented as an integer. It needs to be treated as unsigned
const extendBy = newValidFor - validFor;
const extended = add(expire, { seconds: extendBy });
const payload: Prisma.DataTableUpdateArgs = {
where: { address: addressAsUnsigned },
data: {
...safeProperties,
address: addressAsUnsigned,
expire: extended,
valid: authenticated,
state: {},
},
}
Logger.debug(payload);
// contains the changes I expect
const result = await db.dataTable.update(payload);
Logger.debug(result);
// result contains the same values as before the update.
// And indeed, when I check the database, nothing changed.
// Something like this does what I want, so there is really nothing complicated going on...
await db.$executeRaw`
UPDATE data_table SET
expire = ${extended},
valid = ${authenticated}
WHERE address = ${addressAsUnsigned}
`;
Hopefully, I have not missed something too obvious.
In my experience,
Why is the update (using the ORM way) not performed ?
You might be updating the wrong thing. Is your address an #unique field in your prisma.schema?
Why the value of data.address seems to affect the outcome of the update ?
Prisma might have messed some things up with wrong data. If your data is not unique, you might be updating the first row with that address. If you want to update multiple fields with same address, use updateMany
Do I have to provide all of the fields of the entity when updating ? Or in this case, could I just put what I want changed inside of data ?
No, you only need to put in the data that you need. In your "where" field, add the unique address, and in your data, only the fields that you are changing. In your case, expired and valid. If you want to skip updating some values, use "expired: undefined" and so on
Since you are using typescript, I would advise you to put your object directly inside the prisma update to get the correct types. (At least to fix this problem)
prisma.dataTable.update({where: {...}})
This way you will get the correct types. There is also a command to list all available args inside (control + space on mac)
Note that using the spread operator (...) will remove the listed types, so use it last.
Some other things: double check if your prisma import is correct. Is your data correct, is your returned data correct? Did you refresh your database on update? It might be updated but you just need to refresh for new changes.
I want to call a custom python function on some existing attribute of every document in the entire collection and store the result as a new key-value pair in that (same) document. May I know if there's any way to do that (since each call is independent of others) ?
I noticed cursor.forEach but can't it be done just using python efficiently ?
A simple example would be to split the string in text and store the no. of words as a new attribute.
def split_count(text):
# some complex preprocessing...
return len(text.split())
# Need something like this...
db.collection.update_many({}, {'$set': {"split": split_count('$text') }}, upsert=True)
But it seems like setting a new attribute in a document based on the value of another attribute in the same document is not possible this way yet. This post is old but the issues seem to be still open.
I found a way to call any custom python function on a collection using parallel_scan in PyMongo.
def process_text(cursor):
for row in cursor.batch_size(200):
# Any complex preprocessing here...
split_text = row['text'].split()
db.collection.update_one({'_id': row['_id']},
{'$set': {'split_text': split_text,
'num_words': len(split_text) }},
upsert=True)
def preprocess(num_threads=4):
# Get up to max 'num_threads' cursors.
cursors = db.collection.parallel_scan(num_threads)
threads = [threading.Thread(target=process_text, args=(cursor,)) for cursor in cursors]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
This is not really faster than cursor.forEach (but not that slow either), but it helps me execute any arbitrarily complex python code and save the results from within Python itself.
Also if I have an array of ints in one of the attributes, doing cursor.forEach converts them to floats which I don't want. So I preferred this way.
But I would be glad to know if there're any better ways than this :)
It is quite unlikely that it will ever be efficient to do this kind of thing in python. This is because the document would have to make a round trip and go through the python function on the client machine.
In your example code, you are passing the result of a function to a mongodb update query, which won't work. You can't run any python code inside mongodb queries on the db server.
As the answer to you linked question suggests, this type of action has to be performed in the mongo shell. e.g:
db.collection.find().snapshot().forEach(
function (elem) {
splitLength = elem.text.split(" ").length
db.collection.update(
{
_id: elem._id
},
{
$set: {
split: splitLength
}
}
);
}
);
I'm new to Meteor. I've been stuck on this problem for a while. I can successfully adds items to a collection and look at them fully in the console. However, I cannot access all of the read operations in my .js file.
That is, I can use .find() and .findOne() with empty parameters. But when I try to add .sort or an argument I get an error telling me the object is undefined.
Autopublish is turned on, so I'm not sure what the problem is. These calls are being made directly in the client.
This returns something--
Template.showcards.events({
"click .play-card": function () {
alert(Rounds.find());
}
})
And this returns nothing--
Template.showcards.events({
"click .play-card": function () {
alert(Rounds.find().sort({player1: -1}));
}
})
Sorry for the newbie question. Thanks in advance.
Meteor's collection API works a bit differently from the mongo shell's API, which is understandably confusing for new users. You'll need to do this:
Template.showcards.events({
'click .play-card': function() {
var sortedCards = Rounds.find({}, {sort: {player1: -1}}).fetch();
console.log(sortedCards);
}
});
See this for more details. Also note that logging a cursor (the result of a find) probably isn't what you want. If you want to see the contents of the documents, you need to fetch them.
Rounds.find().sort({player1: -1}) returns a cursor, so you will want to do this:
Rounds.find().sort({player1: -1}).fetch();
Note that this returns an Array of document objects. So you would do something more like this:
docs = Rounds.find().sort({player1: -1}).fetch();
alert(docs[0]);
Im trying to 'compare' all documents between 2 collections, which will return true only and if only all documents inside 2 collections are exactly equal.
I've been searching for the methods on the collection, but couldnt find one that can do this.
I experimented something like these in the mongo shell, but not working as i expected :
db.test1 == db.test2
or
db.test1.to_json() == db.test2.to_json()
Please share your thoughts ! Thank you.
You can try using mongodb eval combined with your custom equals function, something like this.
Your methods don't work because in the first case you are comparing object references, which are not the same. In the second case, there is no guarantee that to_json will generate the same string even for the objects that are the same.
Instead, try something like this:
var compareCollections = function(){
db.test1.find().forEach(function(obj1){
db.test2.find({/*if you know some properties, you can put them here...if don't, leave this empty*/}).forEach(function(obj2){
var equals = function(o1, o2){
// here goes some compare code...modified from the SO link you have in the answer.
};
if(equals(ob1, obj2)){
// Do what you want to do
}
});
});
};
db.eval(compareCollections);
With db.eval you ensure that code will be executed on the database server side, without fetching collections to the client.
I'm making an app similar to instagram using Rails/Mongoid. I want a unique ID that I can use in a url like http://instagr.am/p/DJmU8/
What's the easiest way to do that? Can I derive such an ID from the default BSON ObjectID Mongo creates?
You may try to use first 4 bytes of ObjectID (they will represent timestamp).
But, to be 100% safe, it's better to produce really unique short id, by implementing a counter. You can use separate collection to maintain current value of your counter.
More details on mongo's ObjectID structure can be found here: http://www.mongodb.org/display/DOCS/Object+IDs
As an alternative you can convert convert hex string id representation to a representation based on 36 symbols (26 latin letters + 10 digits). It will obviously be shorter.
It seems, that there is a ruby library, that can do such conversions http://rubyworks.github.com/radix/
Why not use dylang/shortid?
Install using npm npmjs.com/package/shortid:
npm i shortid
Then require:
const shortid = require('shortid');
In mongoose schema:
new Schema {
_id: {
type: String,
default: shortid.generate
}
}
or just insert directly:
users.insert({
_id: shortid.generate()
name: ...
email: ...
});
You could try Mongoid::Token
https://github.com/thetron/mongoid_token
From the docs:
This library is a quick and simple way to generate unique, random
tokens for your mongoid documents, in the cases where you can't, or
don't want to use slugs, or the default MongoDB IDs.
Mongoid::Token can help turn this:
http://myawesomewebapp.com/video/4dcfbb3c6a4f1d4c4a000012/edit
Into something more like this:
http://myawesomewebapp.com/video/83xQ3r/edit
#aav was mention that you can use first 4 bytes, but this value are in seconds and you can get even 10.000 or more insert per seconds. Other thing objectID is Uniq and you need check "when" you get error from duplicate value "Write Concerns"?
new Date().getTime() - is in milliseconds => 1557702577900 why not use last 4 bytes ? Timestamp in base62 is rqiJgPq
This code look interesting:
https://github.com/treygriffith/short-mongo-id/blob/master/lib/objectIdToShortId.js
Check also ObjectID timestamp parser:
https://steveridout.github.io/mongo-object-time/
Or you can execute ObjectId().toString() and base of this string create new by hashids [nodejs,php, andmanymore]
Maybe best options it to use 4-5 bytes from js timestamp and INC from
bson then hash this value by hids
var id = ObjectID('61e33b8467a45920f80eba52').toString();
console.log("id:", id);
console.log("timestamp:", parseInt(id.substring(0, 8),16).toString());
console.log("machineID:", parseInt(id.substring(8, 14),16) );
console.log("processID:", parseInt(id.substring(14, 18),16) );
console.log("counter:", parseInt(id.slice(-6),16) );
var ObjTimestamp = parseInt(ObjectID2.substring(0, 8),16).toString().slice(-5);
var Counter = parseInt(ObjectID2.slice(-6),16);
//https://github.com/base62/base62.js/
console.log('Final:',base62.encode(parseInt(Counter) + parseInt(ObjTimestamp) ));
Output:
vI7o
15V9L
5t4l
You can get collision if: more process are running, then consider add PID to unique and when you run multiple instance on different computer
Try gem https://github.com/jffjs/mongoid_auto_inc
The Hashids library is meant for generating IDs like this. Check it out here ☞ https://github.com/peterhellberg/hashids.rb