I should build a web application to track users activities and I have some issues to understand how can use Redis for tracking online users activities and Mongo to store that data for analyzing it.
I could use just Mongo but I'm worried about the fact I have a lot of invocations to follow what a user is doing. So I was thinking to write on Redis the online data and put in Mongo when they become old. I mean old when the data is meaningless for being online.
I thought about one gateway between Mongo and Redis, so could be RabbitMQ?.
Any suggestions?
I should use just Mongo?
Just an example of code I wrote:
Front-end (Angular application / Socket.io):
setInterval(function () {
socket.emit('visitor-data', {
referringSite: document.referrer,
browser: navigator.sayswho,
os: navigator.platform,
page: location.pathname
});
}, 3000);
Back-end ( Node.js/ Socket.io)
socket.on('visitor-data', function(data) {
visitorsData[socket.id] = data;
);
VisitorsData is a just in an array, but I should build a scalable application, so I can't store data anymore in this way.
Then I have some functions like this for computing the data:
function computeRefererCounts() {
var referrerCounts = {};
for (var key in visitorsData) {
var referringSite = visitorsData[key].referringSite || '(direct)';
if (referringSite in referrerCounts) {
referrerCounts[referringSite]++;
} else {
referrerCounts[referringSite] = 1;
}
}
return referrerCounts;
}
Just some numbers:
I estimated something like :
1 million users for day
15 million activities for day.
Related
Given the data structure below in firebase, i want to run a query to retrieve the blog 'efg'. I don't know the user id at this point.
{Users :
"1234567": {
name: 'Bob',
blogs: {
'abc':{..},
'zyx':{..}
}
},
"7654321": {
name: 'Frank',
blogs: {
'efg':{..},
'hij':{..}
}
}
}
The Firebase API only allows you to filter children one level deep (or with a known path) with its orderByChild and equalTo methods.
So without modifying/expanding your current data structure that just leaves the option to retrieve all data and filter it client-side:
var ref = firebase.database().ref('Users');
ref.once('value', function(snapshot) {
snapshot.forEach(function(userSnapshot) {
var blogs = userSnapshot.val().blogs;
var daBlog = blogs['efg'];
});
});
This is of course highly inefficient and won't scale when you have a non-trivial number of users/blogs.
So the common solution to that is to a so-called index to your tree that maps the key that you are looking for to the path where it resides:
{Blogs:
"abc": "1234567",
"zyx": "1234567",
"efg": "7654321",
"hij": "7654321"
}
Then you can quickly access the blog using:
var ref = firebase.database().ref();
ref.child('Blogs/efg').once('value', function(snapshot) {
var user = snapshot.val();
ref.child('Blogs/'+user+'/blogs').once('value', function(blogSnapshot) {
var daBlog = blogSnapshot.val();
});
});
You might also want to reconsider if you can restructure your data to better fit your use-case and Firebase's limitations. They have some good documentation on structuring your data, but the most important one for people new to NoSQL/hierarchical databases seems to be "avoid building nests".
Also see my answer on Firebase query if child of child contains a value for a good example. I'd also recommend reading about many-to-many relationships in Firebase, and this article on general NoSQL data modeling.
Given your current data structure you can retrieve the User that contains the blog post you are looking for.
const db = firebase.database()
const usersRef = db.ref('users')
const query = usersRef.orderByChild('blogs/efg').limitToLast(1)
query.once('value').then((ss) => {
console.log(ss.val()) //=> { '7654321': { blogs: {...}}}
})
You need to use limitToLast since Objects are sorted last when using orderByChild docs.
It's actually super easy - just use foreslash:
db.ref('Users').child("userid/name")
db.ref('Users').child("userid/blogs")
db.ref('Users').child("userid/blogs/abc")
No need of loops or anything more.
I have a chat app, that is using Ionic 2 and Meteor with MongoDB. It works perfectly.
However, everything is stored in the MongoDB on the server, so each time a user wants to view their messages, they need to be connected to the Meteor/Mongo Server running in the cloud. Also, if one user deletes their chat, it will delete the chat on the MongoDB, and the corresponding other user will also have their chat deleted.
I would like similar functionality as WhatsApp where the messages are held locally on the device (I am using SQLite), and only new messages are held in the cloud until both users download them.
Currently my app iterates over a Mongo.Cursor<Chat> object. It also observes this object (this.chats.observe({changed: (newChat, oldChat) => this.disposeChat(oldChat), removed: (chat) => this.disposeChat(chat)});).
I get chat data from SQLlite that I have stored locally (Array<Chat>).
Question
Is it possible to add the SQLite data (Array<Chat>) to the Mongo.Cursor<Chat>? When I do so, I want to just add to minimongo and not MongoDB on the server.
Thanks
UPDATE
Asp per advise below, I do the following:
let promise: Promise<Mongo.Cursor<Chat>> = new Promise<Mongo.Cursor<Chat>>(resolve => {
this.subscribe('chats', this.senderId, registeredIds, () => {
let chats: Mongo.Cursor<Chat> = Chats.find(
{ memberIds: { $in: registeredIds } },
{
sort: { lastMessageCreatedAt: -1 },
transform: this.transformChat.bind(this),
fields: { memberIds: 1, lastMessageCreatedAt: 1 }
}
);
this.localChatCollection = new Mongo.Collection<Chat>(null);
console.log(this.localChatCollection);
chats.forEach(function (chat: Chat) {
console.log('findChats(): add chat to collection: ' + chat);
this.localChatCollection.insert(chat);
});
Will update if it works.
UPDATE
When I do the following, it inserts the chat object:
let promise: Promise<Mongo.Collection<Chat>> = this.findChats();
promise.then((data: Mongo.Collection<Chat>) => {
let localChatCollection: Mongo.Collection<Chat> = new Mongo.Collection<Chat>(null);
data.find().forEach(function (chat: Chat) {
console.log('==> ' + chat);
localChatCollection.insert(chat);
});
However, if I define the localChatCollection globally, it does not insert the chat object. There are no errors but the process just stops on the insert line.
private localChatCollection: Mongo.Collection<Chat> = new Mongo.Collection<Chat>(null);
....
this.localChatCollection.insert(chat);
Any ideas how I can get this to insert into a globally defined collection?
Is it possible to add the SQLite data (Array) to the Mongo.Cursor? When I do so, I want to just add to minimongo and not MongoDB on the server.
Meteor itself knows nothing about SQLite, but it sounds like you have that part of it working.
To just add to minimongo and not the mongodb server, you're looking for a client-side collection. Just pass null in as the first parameter to the call to create your collection i.e.
var localChatCollection = new Mongo.Collection(null)
You can then insert to localChatCollection the same way you would with a synchronized collection.
Source: Meteor docs
Ok, still in my toy app, I want to find out the average mileage on a group of car owners' odometers. This is pretty easy on the client but doesn't scale. Right? But on the server, I don't exactly see how to accomplish it.
Questions:
How do you implement something on the server then use it on the client?
How do you use the $avg aggregation function of mongo to leverage its optimized aggregation function?
Or alternatively to (2) how do you do a map/reduce on the server and make it available to the client?
The suggestion by #HubertOG was to use Meteor.call, which makes sense and I did this:
# Client side
Template.mileage.average_miles = ->
answer = null
Meteor.call "average_mileage", (error, result) ->
console.log "got average mileage result #{result}"
answer = result
console.log "but wait, answer = #{answer}"
answer
# Server side
Meteor.methods average_mileage: ->
console.log "server mileage called"
total = count = 0
r = Mileage.find({}).forEach (mileage) ->
total += mileage.mileage
count += 1
console.log "server about to return #{total / count}"
total / count
That would seem to work fine, but it doesn't because as near as I can tell Meteor.call is an asynchronous call and answer will always be a null return. Handling stuff on the server seems like a common enough use case that I must have just overlooked something. What would that be?
Thanks!
As of Meteor 0.6.5, the collection API doesn't support aggregation queries yet because there's no (straightforward) way to do live updates on them. However, you can still write them yourself, and make them available in a Meteor.publish, although the result will be static. In my opinion, doing it this way is still preferable because you can merge multiple aggregations and use the client-side collection API.
Meteor.publish("someAggregation", function (args) {
var sub = this;
// This works for Meteor 0.6.5
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
// Your arguments to Mongo's aggregation. Make these however you want.
var pipeline = [
{ $match: doSomethingWith(args) },
{ $group: {
_id: whatWeAreGroupingWith(args),
count: { $sum: 1 }
}}
];
db.collection("server_collection_name").aggregate(
pipeline,
// Need to wrap the callback so it gets called in a Fiber.
Meteor.bindEnvironment(
function(err, result) {
// Add each of the results to the subscription.
_.each(result, function(e) {
// Generate a random disposable id for aggregated documents
sub.added("client_collection_name", Random.id(), {
key: e._id.somethingOfInterest,
count: e.count
});
});
sub.ready();
},
function(error) {
Meteor._debug( "Error doing aggregation: " + error);
}
)
);
});
The above is an example grouping/count aggregation. Some things of note:
When you do this, you'll naturally be doing an aggregation on server_collection_name and pushing the results to a different collection called client_collection_name.
This subscription isn't going to be live, and will probably be updated whenever the arguments change, so we use a really simple loop that just pushes all the results out.
The results of the aggregation don't have Mongo ObjectIDs, so we generate some arbitrary ones of our own.
The callback to the aggregation needs to be wrapped in a Fiber. I use Meteor.bindEnvironment here but one can also use a Future for more low-level control.
If you start combining the results of publications like these, you'll need to carefully consider how the randomly generated ids impact the merge box. However, a straightforward implementation of this is just a standard database query, except it is more convenient to use with Meteor APIs client-side.
TL;DR version: Almost anytime you are pushing data out from the server, a publish is preferable to a method.
For more information about different ways to do aggregation, check out this post.
I did this with the 'aggregate' method. (ver 0.7.x)
if(Meteor.isServer){
Future = Npm.require('fibers/future');
Meteor.methods({
'aggregate' : function(param){
var fut = new Future();
MongoInternals.defaultRemoteCollectionDriver().mongo._getCollection(param.collection).aggregate(param.pipe,function(err, result){
fut.return(result);
});
return fut.wait();
}
,'test':function(param){
var _param = {
pipe : [
{ $unwind:'$data' },
{ $match:{
'data.y':"2031",
'data.m':'01',
'data.d':'01'
}},
{ $project : {
'_id':0
,'project_id' : "$project_id"
,'idx' : "$data.idx"
,'y' : '$data.y'
,'m' : '$data.m'
,'d' : '$data.d'
}}
],
collection:"yourCollection"
}
Meteor.call('aggregate',_param);
}
});
}
If you want reactivity, use Meteor.publish instead of Meteor.call. There's an example in the docs where they publish the number of messages in a given room (just above the documentation for this.userId), you should be able to do something similar.
You can use Meteor.methods for that.
// server
Meteor.methods({
average: function() {
...
return something;
},
});
// client
var _avg = { /* Create an object to store value and dependency */
dep: new Deps.Dependency();
};
Template.mileage.rendered = function() {
_avg.init = true;
};
Template.mileage.averageMiles = function() {
_avg.dep.depend(); /* Make the function rerun when _avg.dep is touched */
if(_avg.init) { /* Fetch the value from the server if not yet done */
_avg.init = false;
Meteor.call('average', function(error, result) {
_avg.val = result;
_avg.dep.changed(); /* Rerun the helper */
});
}
return _avg.val;
});
I'm trying to use as much of the OOTB sync and RESTful functionality in Backbone. I have a Web API set up for basic CRUD for my models. I have:
var SearchModel = Backbone.Model.extend({});
var SearchMappingModel = Backbone.Model.extend({});
var SearchComponentModel = Backbone.Model.extend({});
var SearchCollection = Backbone.Collection.extend({});
var SearchMappingCollection = Backbone.Collection.extend({});
var SearchComponentCollection = Backbone.Collection.extend({});
For every Search there is 1-to-many SearchMappings, and for every SearchMapping, there are 1-to-many SearchComponents. My URLs for sync would be something like, "/search" for the Search collection, "'/searchmapping/' + searchId" for the SearchMapping collection, and "'/searchcomponent/' + mappingId" for the SearchComponent collection.
My question is, since each collection is dependent on the previous one, is there a way I can make a cascading relationship in backbone to minimize my code and use as much of the basic sync functionality that's already there?
My initial thought is to create a collection within a collection and write my own .fetch() to first fetch the parent collection and on its success then fetch the child, which will then also get its child after its own success, like this:
var SearchCollection = Backbone.Collection.extend({
model: SearchModel,
initialize: function (data) {
this.url = baseURL + "/search";
this.data = data;
this.SearchMappingCollection = new SearchMappingCollection();
},
fetchData: function () {
this.fetch({
success: _.bind(function (results) {
this.fetchListSuccess(results);
}, this)
});
},
fetchListSuccess: function (results) {
this.SearchMappingCollection.fetchData(results);
}
The same would be done on a .save(). This may be a good way of doing it, but wanted to get feedback from anyone else that's done something similar.
I ended up not using a cascading format. It seems that it was adding more complexity and giving nothing in return. All 3 collections now sit on the controller level, and I just load the next collection after each collection is loaded on each "reset" event.
I'm looking to learn node.js and mongodb which look suitable for something I'd like to make. As a little project to help me learn I thought I'd copy the "posts" table from a phpbb3 forum I have into a mongodb table so I did something like this where db is mongodb database connection, and client is a mysql database connection.
db.collection('posts', function (err, data) {
client.query('select * from phpbb_posts", function(err, rs) {
data.insert(rs);
});
this works ok when I do it on small tables, but my posts table has about 100000 rows in and this query doesn't return even when I leave it running for an hour. I suspect that it's trying to load the entire database table into memory and then insert it.
So what I would like to do is read a chunk of rows at a time and insert them. However I can't see how to read a subset of the rows in node.js, and even more of a problem, I can't understand how I can iterate through the queries one at a time when I only get notification via a callback that it's finished.
Any ideas how I can best do this? (I'm looking for solutions using node.js as I'd like to know how to solve this kind of problem, I could no doubt do it easily some other way)
You could try using the asnyc library by caolan. The library implements some async flow control methods to handle the caveats of a callback-oriented programming style as it is in node.js.
For your case, using the whilst method could work out, using LIMIT queries against mysql and inserting them into mongodb.
Example (not tested, as i have no testdata available, but i think you'll get the idea)
var insertCount = 0;
var offset = 0;
// set this to the overall recordcound from mysql
var recordCount = 0;
async.whilst(
// test condition callback
function () { return insertCount < recordCount; },
// actual worker callback
function (callback) {
db.collection('posts', function (err, data) {
client.query('select * from phpbb_posts LIMIT ' + insertCount + ',1000', function(err, rs) {
data.insert(rs);
// increment by actually fetched recordcount (res.length?)
insertCount += res.length;
// trigger flow callback
callback();
});
});
},
// finished callback
function (err) {
// finished inserting data, maybe check record count in mongodb here
}
});
As i already mentioned, this code is just adapted from an example of the async library readme. But maybe it is an option for adding such amounts of database records from mysql to mongo.