I'm having a issue where my mongo collection is behaving differently after I've uploaded it to Meteor's servers. Locally everything works perfectly and I'm not seeing any issues when running meteor logs either.
What I'm trying to do is this:
In my collection called RaceList I have several entry's. Each has a unique id, an array of users and a 'live' variable which is a boolean.
Every hour I update this collection by removing the live race, setting the next race's live variable to true and adding another race to the end of the collection.
All this is working for me locally but after uploading to my Meteor server something strange is happening. If I don't and anybody to the array of users in the next race to turn live it seems to be working ok but as soon as I join the race, for some reason, the race immediately after the one I have just joined will become the live race and the race I joined is skipped...
Here is the code from my server that is executed every hour:
updateRaces: ->
# Remove the finished race
Meteor.call 'removeLiveRace'
# Set the next race to live
Meteor.call 'updateLiveRace'
# Add another race to the collection
Meteor.call 'insertNewRace'
And here is the code from my Meteor.methods
removeLiveRace: ->
id = RaceList.findOne( { live: true } )?._id
if id
RaceList.remove _id: id
updateLiveRace: ->
id = _.first( RaceList.find().fetch() )._id
RaceList.update id, $set: live: true
insertNewRace: ->
RaceList.insert
live : false
users : []
Any help is greatly appreciated. I'm still just getting started with Meteor so any advice to make this code more efficient/safe would be great!
Thanks : )
While it doesn't look like you're using Cron, the standard warning message still applies to you:
Caveats
Beware, SyncedCron probably won't work as expected on certain shared
hosting providers that shutdown app instances when they aren't
receiving requests (like Heroku's free dyno tier or Meteor free
galaxy).
In other words, because you are using free services, any moments of app shutdown will mess with any Cron/time-based functions.
Figured it out. It was because my collection was sending an entry to the bottom of the collection when I added more than 1 user to it. I have no idea why this would happen? I just set the collection to sort by createdAt wherever I query it and that fixed the problem.
Related
Our Java app saves its configurations in a MongoDB collections. When the app starts it reads all the configurations from MongoDB and caches them in Maps. We would like to use the change stream API to be able also to watch for updates of the configurations collections.
So, upon app startup, first we would like to get all configurations, and from now on - watch for any further change.
Is there an easy way to execute the following atomically:
A find() that retrieves all configurations (documents)
Start a watch() that will send all further updates
By atomically I mean - without potentially missing any update (between 1 and 2 someone could update the collection with new configuration).
To make sure I lose no update notifications, I found that I can use watch().startAtOperationTime(serverTime) (for MongoDB of 4.0 or later), as follows.
Query the MongoDB server for its current time, using command such as Document hostInfoDoc = mongoTemplate.executeCommand(new Document("hostInfo", 1))
Query for all interesting documents: List<C> configList = mongoTemplate.findAll(clazz);
Extract the server time from hostInfoDoc: BsonTimestamp serverTime = (BsonTimestamp) hostInfoDoc.get("operationTime");
Start the change stream configured with the saved server time ChangeStreamIterable<Document> changes = eventCollection.watch().startAtOperationTime(serverTime);
Since 1 ends before 2 starts, we know that the documents that were returned by 2 were at least same or fresher than the ones on that server time. And any updates that happened on or after this server time will be sent to us by the change stream (I don't care to run again redundant updates, because I use map as cache, so extra add/remove won't make a difference, as long as the last action arrives).
I think I could also use watch().resumeAfter(_idOfLastAddedDoc) (didn't try). I did not use this approach because of the following scenario: the collection is empty, and the first document is added after getting all (none) documents, and before starting the watch(). In that scenario I don't have previous document _id to use as resume token.
Update
Instead of using "hostInfo" for getting the server time, which couldn't be used in our production, I ended using "dbStats" like that:
Document dbStats= mongoOperations.executeCommand(new Document("dbStats", 1));
BsonTimestamp serverTime = (BsonTimestamp) dbStats.get("operationTime");
I have a server running IP controller and 12 IPengines. I connect to the controller from my laptop using SSH. I submitted some jobs to the controller using the load-balanced view interface (in non-blocking mode) and stored the message IDs in the Asyc Result object returned the by apply_async() method.
I accidentally lost the message IDs for the jobs and wanted to know if there's a way to retrieve the job IDs (or the results) from the Hub database. I use a SQLite database for the Hub, and I can get the rc.db_query() method to work, but I don't know what to look for.
Does anyone know how to query the Hub database only for message IDs of the jobs I submitted? What's the easiest way of retrieving the job results from the Hub, if I don't have access to the AsyncHubResult object (or their message IDs)?
Thanks!
Without the message IDs, you are might have a pretty hard time finding the right tasks, unless there haven't been so many tasks submitted.
The querying is based on MongoDB (it's a passthrough when you use mongodb, and a subset of simple operators are implemented for sqlite).
Quick summary: a query is a dict. If you use literal values, they are equality tests, but you can use dict values for comparison operators.
You can search by date for any of the timestamps:
submitted: arrived at the controller
started: arrived on an engine
completed: finished on the engine
For instance, to find tasks submitted yesterday:
from datetime import date, time, timedelta, datetime
# round to midnight
today = datetime.combine(date.today(), time())
yesterday = today - timedelta(days=1)
rc.db_query({'submitted': {
'$lt': today, # less than midnight last night
'$gt': yesterday, # greater than midnight the night before
}})
or all tasks submitted 1-4 hours ago:
found = rc.db_query({'submitted': {
'$lt': datetime.now() - timedelta(hours=1),
'$gt': datetime.now() - timedelta(hours=4),
}})
With the results of that, you can look at keys like client_uuid to retrieve all messages submitted by a given client instance (e.g. a single notebook or script):
client_id = found[0]['client_uuid']
all_from_client = rc.db_query({'client_uuid': client_uuid})
Since you are only interested in results at this point, you can specify keys=['msg_id'] to only retrieve the message IDs. We can then use these msg_ids to get all the results produced by a single client session:
# construct list of msg_ids
msg_ids = [ r['msg_id'] for r in rc.db_query({'client_uuid': client_uuid}, keys=['msg_id']) ]
# use client.get_result to retrieve the actual results:
results = rc.get_result(msg_ids)
At this point, you have all of the results, but you have lost the association of which results came from which execution. There isn't a lot of info to help you out there, but you might be able to tell by type, timestamps, or perhaps select the 9 final items from a given session.
I'm working on a filtered live search module with Meteor.js.
Usecase & problem:
A user wants to do a search through all the users to find friends. But I cannot afford for each user to ask the complete users collection. The user filter the search using checkboxes. I'd like to subscribe to the matched users. What is the best way to do it ?
I guess it would be better to create the query client-side, then send it the the method to get back the desired set of users. But, I wonder : when the filtering criteria changes, does the new subscription erase all of the old one ? Because, if I do a first search which return me [usr1, usr3, usr5], and after that a search that return me [usr2, usr4], the best would be to keep the first set and simply add the new one to it on the client-side suscribed collection.
And, in addition, if then I do a third research wich should return me [usr1, usr3, usr2, usr4], the autorunned subscription would not send me anything as I already have the whole result set in my collection.
The goal is to spare processing and data transfer from the server.
I have some ideas, but I haven't coded enough of it yet to share it in a easily comprehensive way.
How would you advice me to do to be the more relevant possible in term of time and performance saving ?
Thanks you all.
David
It depends on your application, but you'll probably send a non-empty string to a publisher which uses that string to search the users collection for matching names. For example:
Meteor.publish('usersByName', function(search) {
check(search, String);
// make sure the user is logged in and that search is sufficiently long
if (!(this.userId && search.length > 2))
return [];
// search by case insensitive regular expression
var selector = {username: new RegExp(search, 'i')};
// only publish the necessary fields
var options = {fields: {username: 1}};
return Meteor.users.find(selector, options);
});
Also see common mistakes for why we limit the fields.
performance
Meteor is clever enough to keep track of the current document set that each client has for each publisher. When the publisher reruns, it knows to only send the difference between the sets. So the situation you described above is already taken care of for you.
If you were subscribed for users: 1,2,3
Then you restarted the subscription for users 2,3,4
The server would send a removed message for 1 and an added message for 4.
Note this will not happen if you stopped the subscription prior to rerunning it.
To my knowledge, there isn't a way to avoid removed messages when modifying the parameters for a single subscription. I can think of two possible (but tricky) alternatives:
Accumulate the intersection of all prior search queries and use that when subscribing. For example, if a user searched for {height: 5} and then searched for {eyes: 'blue'} you could subscribe with {height: 5, eyes: 'blue'}. This may be hard to implement on the client, but it should accomplish what you want with the minimum network traffic.
Accumulate active subscriptions. Rather than modifying the existing subscription each time the user modifies the search, start a new subscription for the new set of documents, and push the subscription handle to an array. When the template is destroyed, you'll need to iterate through all of the handles and call stop() on them. This should work, but it will consume more resources (both network and server memory + CPU).
Before attempting either of these solutions, I'd recommend benchmarking the worst case scenario without using them. My main concern is that without fairly tight controls, you could end up publishing the entire users collection after successive searches.
If you want to go easy on your server, you'll want to send as little data to the client as possible. That means every document you send to the client that is NOT a friend is waste. So let's eliminate all that waste.
Collect your filters (eg filters = {sex: 'Male', state: 'Oregon'}). Then call a method to search based on your filter (eg Users.find(filters). Additionally, you can run your own proprietary ranking algorithm to determine the % chance that a person is a friend. Maybe base it off of distance from ip address (or from phone GPS history), mutual friends, etc. This will pay dividends in efficiency in a bit. Index things like GPS coords or other highly unique attributes, maybe try out composite indexes. But remember more indexes means slower writes.
Now you've got a cursor with all possible friends, ranked from most likely to least likely.
Next, change your subscription to match those friends, but put a limit:20 on there. Also, only send over the fields you need. That way, if a user wants to skip this step, you only wasted sending 20 partial docs over the wire. Then, have an infinite scroll or 'load more' button the user can click. When they load more, it's an additive subscription, so it's not resending duplicate info. Discover Meteor describes this pattern in great detail, so I won't.
After a few clicks/scrolls, the user won't find any more friends (because you were smart & sorted them) so they will stop trying & move on to the next step. If you returned 200 possible friends & they stop trying after 60, you just saved 140 docs from going through the pipeline. There's your efficiency.
So I've worked my way through an evening with ChicagoBoss. I am now currently trying to wire up ChicagoBoss with MongoDB to build an app with it (and learn two new technologies I've been eyeing in the process). Unfortunately I seem to have hit a snag. Specifically, after a user is created, the password does not appear to be what it was set to. Please note that with the exception of configuration code, all code comes directly from the Chicago Boss tutorials.
boss.config - re Databases:
{db_host, "localhost"},
{db_port, 27017},
{db_adapter, mongodb},
login function
login('POST', []) ->
Name = Req:post_param("name"),
case boss_db:find(user, [{name, Name}]) of
[User] ->
case User:check_password(Req:post_param("password")) of
true ->
{redirect, proplists:get_value("redirect",
Req:post_params(), "/"), User:login_cookies()};
false ->
{ok, [{error, "Bad name/password combination"}]}
end;
[] ->
{ok, [{error, "No User named " ++ Name}]}
end.
Then - after creating a user from the admin interface and using
hash_for(Name, Password) ->
Salt = mochihex:to_hex(erlang:md5(Name)),
hash_password(Password, Salt).
to generate a hash, the following two things happen:
1) The user is created but not id-ed as user-1, but rather as usr-51970a2a3e01c027d4000001.
Why is that? I thought ChicagoBoss followed the rails convention of autoincrementing numeric indices?
2) Even though the password that was passed into hash_for is used, the user can never login.
3) After the Chicago Boss server is restarted the newly created database objects are retained - so mongo is at least saving something correctly.
Can anyone shed any light on whats going on here? How is MongoDB trying to save the user? What is the correct method for wiring up Chicago Boss with MongoDB, does boss_db not handle MongoDB natively?
The tutorial all of this is trying to use as a jump off point is:
https://github.com/evanmiller/ChicagoBoss/wiki/An-Evening-With-Chicago-Boss
MongoDB is indeed supported in Chicago Boss, but some things will be different with MongoDB as compared to a SQL database. In particular, because it is "humongous", MongoDB uses UUIDs instead of auto-incrementing IDs. This allows the creation of new documents without keeping a centralized counter. So that explains why the ID is "usr-51970a2a3e01c027d4000001" and not "usr-1".
As for the password issue, I think the next step is to add debug printing statements. What hash value is the DB expecting? Is the DB saving the correct hash value? If you created the user through the admin interface there is potential for slight errors such as an extra newline in the password.
Also, feel free to ping us on the mailing list or on IRC at any time.
we have a button in a web game for the users to collect reward. That should only be clicked once, and upon receiving the request, we'll mark it collected in DB.
we've already blocked the buttons in the client from repeated clicking. But that won't help if people resend the package multiple times to our server in short period of time.
what I want is a method to block this from server side.
we're using Playframework 2 (2.0.3-RC2) for server side and so far it's stateless, I'm tempted to use a Set to guard like this:
if processingSet has userId then BadRequest
else put userId in processingSet and handle request
after that remove userId from that Set
but then I'd have to face problem like Updating Scala collections thread-safely and still fail to block the user once we have more than one server behind load balancing.
one possibility I'm thinking about is to have a table in DB in place of the processingSet above, but that would incur 1+ DB operation per request, are there any better solution~?
thanks~
Additional DB operation is relatively 'cheap' solution in that case. You should use it if you'e planning to save the buttons state permanently.
If the button is disabled only for some period of time (for an example until the game is over) you can also consider using the cache API however keep in mind that's not dedicated for solutions which should be stored for long time (it should not be considered as DB alternative).
Given that you're using Mongo and so don't have transactions spanning separate collections, I think you can probably implement this guard using an atomic operation - namely "Update if current", which is effectively CompareAndSwap.
Assuming you've got a collection like "rewards" which has a "collected" attribute, you can update the collected flag to true only if it is currently false and if that operation doesn't fail you can proceed to apply the reward knowing that for any other requests the same operation will fail.