I'm trying to build a query that first lets me get a list of followers that are following a user, second it should take that list and then check to see if they are 'online'.
I have two 'indexes' or endpoints /channel and /following.
The channel endpoint JSON object looks like this (parts abbreviated)
{ channel: {"username":"username1", ... , "online":"true" } }
The following endpoint object looks a bit like this
{ following : {"username1":{"username2":"username2", "username3":"username3"} }
if I run a simple query /following/_search I get back hits like...
{
"_index": "following",
"_type": "following",
"_id": "_Liso_",
"_score": 1,
"_source": {
"Gabe": "Gabe",
"Gavin": "Gavin"
}
}
This result means that Gavin is following Gabe.
I believe the issue is how I'm storing the data.
In firebase my data looks like this
following
|---Gabe
|----Gavin:Gavin
so each child object of following object has key/value children of {username}:{username}
Now I can run queries that individually get the results I need. For example, if I ask ElasticSearch (ES) if channel "Gavin" is "online" I get back one result depending on if they are or are not online. And same with Following. However I can't seem to get the query to first see who is following Gavin and then see if they are online and return those users whom are online.
I've found a better solution (or maybe not). First you query the database for users whom are following a user.
From this list you send another query
{
"query":{
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"bool":{
"must":{
"terms":{
"username":["username1"]
}
},
"must_not":{
"terms":{
"online":["true"]
}
}
}
}
}
}
}
This works however the username cannot be mixed with capitols. I don't know if this is an indexing issue on my part or terms have to be very specific. The solution I'm using on the client side is to lowercase the search terms before I submit them. It's crude and hacky but it works for now.
Issues that I may run into:
If a user has millions of followers pulling all that data from the
database will make the client sluggish.
a possible solution to this is to paginate the following results and run the query for every 20 returned results.
I'll continue to revise the answer as I develop / learn better query methods.
Related
Problem
I want to retrieve specific data from Google Firestore.
It's only possible to get all of the 'Fields' data. But no specific data within fields
Example of the GET-Request:
https://firestore.googleapis.com/v1beta1/projects/edubox-49528/databases/(default)/documents/nodes/EduBox-1234567?key=[My_API_KEY]&fields=fields
As you can see, It is possible to obtain all the items in the object 'Fields'. But it is not possible to get any further into detail to obtain more specific data (test, message, nodeID, ...)
Tryouts
I have already tried:
fields=fields/test
fields=fields.test
fields=fields(test)
fields=fields/test/integerValue
...
Expected Results
I want to obtain specific data like the String / integer value of my objects in 'Fields'.
This example should return the integerValue with 30
https://firestore.googleapis.com/v1beta1/projects/edubox-49528/databases/(default)/documents/nodes/EduBox-1234567?key=[My_API_KEY]&fields=fields/test
This example should return 30
https://firestore.googleapis.com/v1beta1/projects/edubox-49528/databases/(default)/documents/nodes/EduBox-1234567?key=[My_API_KEY]&fields=fields/test/integerValue
Solution
While browsing the web, I came across Google Api Explorer:
https://developers.google.com/apis-explorer/#search/firestore/firestore/v1beta1/
When trying out some possibilities, I came across this:
https://firestore.googleapis.com/v1/projects/edubox-49528/databases/(default)/documents/nodes/EduBox-1234567?mask.fieldPaths=nodeID&fields=fields&key={YOUR_API_KEY}
This gives me the right information
but I still need a more detailed form of this answer like only the 'EduBox-1234567'
The way to retrieve a specific field is to use mask.fieldPaths. For example the following GET method:
https://firestore.googleapis.com/v1beta1/projects/edubox-49528/databases/(default)/documents/nodes/EduBox-1234567?key=[My_API_KEY]&fields=fields&mask.fieldPaths=nodeID
is going do return:
{
"fields": {
"nodeID": {
"stringValue": "EduBox-1234567"
}
}
Documentation references here and here.
I'm adding custom data to Meteor user accounts for the first time. I've been able to add custom fields without difficulty and I know they're there because I can see them in Mongol. I am publishing via a global subscription so how do I then go about reading data from individual fields? It seems the syntax is very different from that when using publish/subscribe methods.
So, I have user accounts like this (as seen in Mongol):
"_id": "#################",
"profile": {
"name": "Test User"
},
"customfields": {
"customfield1": [
"A","B","C"
]
}
}
In server/main.js I have the following
Meteor.publish(null, function() {
return Meteor.users.find(this.userId, {fields:{customfields:1}});
});
This seems to be publishing fine. But what code do I use to render the cursor as data? I've been using variations on code like this in client/main.js and having no success:
var stuff = Meteor.users.find(this.userId).fetch();
console.log(stuff.customfield1);
Any help appreciated.
MyCollection.find() returns a cursor whereas MyCollection.findOne() returns an object, i.e. a single mongodb document.
A publication must return a cursor or array of cursors. You publication is fine.
You are basically trying to make the customfields key of the user object visible on the client. (The profile key is automatically published by Meteor).
On the client, where you are doing:
var stuff = Meteor.users.find(this.userId).fetch();
You can simply use:
var stuff = Meteor.user();
or
var stuff = Meteor.users.findOne(Meteor.userId());
Then stuff.customfields will contain what you're looking for.
The second form is way too verbose for me unless you're looking for a different user than the logged in user.
Note: this.userId on the client will not be the userId of the current user, it will be undefined. That only works on the server. That may actually be the root cause of your problem. In addition, your publications must be ready() for the data to be available. This isn't true immediately after login for example.
Since customfield1 is nested in customfields, did you try stuff.customfields.customfield1?
First - fair warning for any new user, stay away from appcelerator, that team should be ashamed of the work they have performed. Read on for an example. If I am wrong I will happily admit it. The customer's data already exists and mobile application is in production using existing field names.
I am performing batch_create with (paid) account on arrowdb against their arrow cloud data source. Single queries all work well so I know there are no key or general connectivity issues. Login works, I can do manual or automatic session mgmt and all works well for queries, creates, deletes, etc.
I am using the node arrowdb 1.06 library.
Now we get to batch_create which my code looks like this after a login:
arrowDBApp.post("/v1/objects/EM/batch_create.json",
{"json_array": [
{
old_id: '537280ca1316e90db2335bf7',
old_acl_id: '5372800e1316e90da8331dac',
idp_id: '17',
reference: '17',
serves: [ '52a7c0a0534b2b0b5000002b' ],
meter: 'NO METER',
amps: '110',
property_id: '52a7c02612f73e0b1e0032f7'
}
]},
function (err, result) {
if (err) {
console.error(err.message);
}
else {
console.log(result.body.meta);
}
}
);
With this code I get this error:
Request returned with HTTP status code 400 Invalid object at index 0: Invalid BSON id 17
And if I change the field name "idp_id" to "idpid" the problem goes away and the batch is created. Or, if I change the content from "17" to any GUID "xxxxxxxxxxxxxxxxxxxxxxxx" format.
So the question is, does anyone know a way to have "_id" in a field name without the API validating the content as having to be in the GUID format? Like I said the data exists already and we do not want to change field names and have to update the client apps (web app, mobile app, etc.).
Why would they simply validate any field with '_id'in it as a guid? They don't do it on single creates? I have hundreds of thousands of objects that have to be inserted and would like to batch create them.
thanks for any assistance. Never had anything but poor experience with ACS, always go AWS folks....
Hopefully this question is not too long but I am trying to include as much details as possible in what I did..
I am trying to figure out how to implement logic in Meteor.publish() that takes data from the DB, changes all the values in a column and makes the updated collection available for client-side subscription.
Specifically, I have a table that stores messages between users and the recipient is identified by his userId. I would like to replace the userId with his actual phone number which should be available in the Meteor.users table.
When I looked it up online I saw suggestions to use transform but my understanding is that it's not reactive.. I then learned about map but discovered that it returns an array which breaks the Meteor.publish() method. Finally I found something that uses forEach and self.added() and self.ready() so my code currently looks like this:
Meteor.publish("myMessages", function () {
var self = this;
Messages.find({
$or: [
{ senderId: this.userId },
{ recipientId: this.userId }
]
}).forEach(function(m) {
m.recipientId = Meteor.users.findOne({ _id: m.recipientId }).username;
console.log("adding msg to collection:");
console.log(m);
self.added("Messages", m._id, m);
});
self.ready();
});
The log messages look right and when Meteor restarts it prints all the messages from the DB related to the user where the recipient is replaced correctly with the phone number. However, on the client side when I try to run Messages.findOne(msgId) (with an id I verified exists by selecting it directly in mongo shell) I get undefined back and furthermore, running Messages.find() through developer tools in the browser returns undefined as well although I expected the messages that showed up in the logs to be available..
I feel that this is a basic use case but I am not able to make this work.. any help is appreciated!
"You can transform a collection on the server side like this:"
https://stackoverflow.com/a/18344597/4023641
It worked for me.
Unfortunately, changes in users collection will not update reactively these custom fields.
We have two document 'types': Post and User:
Typical post:
{
"_id": "3847345345",
"Schema": "Post",
"Text": "Hello World! This is a post!",
"IsFeatured": true,
"UserID": "12345345345234234"
}
Typical user:
{
"_id": "12345345345234234",
"Schema": "User",
"Username": "georgepowell"
"PostIds": ["3847345345","5135345345","9987453236", ... ]
}
On a web page that displays a Post, the Username for that post (plus whatever other changable information about that user) is displayed alongside the post. Similar to SO:
This is a typical example of a situation where an SQL JOIN would be perfect, but of course CouchDB doesn't support anything like that. Instead we could make a view that indexes both User documents and Post documents on a Post's _id. Like this:
function(doc) {
if (doc.Schema = 'Post') {
emit([doc._id, 0], null);
} else if (doc.Schema = 'User') {
foreach (string id in doc.PostIds) // not javascript I know. shhh
emit([id, 1], null);
}
}
which works well, as we can efficiently retrieve all the information we need given a single Post's _id.
However, if I want to create a view that lists all the posts where IsFeatured == true along with all the user data, I get stuck!
function(doc) {
if (doc.Schema = 'Post' && doc.IsFeatured) {
emit([doc._id, 0], null);
} else if (doc.Schema = 'User') {
foreach (string id in doc.PostIds)
emit([id, 1], null); // I can't check if the post is featured!
}
}
Have I reached the limit of CouchDB for relational data? or is this kind of indexing possible in CouchDB?
Since it is a different technology there are trade-offs. And sometimes although things look like they will take more resources (an extra request) in the short-run it can be inconsequential, and in the long-run may give significant scalability, if you need that sort of thing.
CouchDB can handle a lot of different "databases" at the same time, which you can think of as different model spaces. So with the same running instance of CouchDB you could have /users and /posts. This requires absolutely no additional work on the part of configuration or performance of CouchDB.
This can make your map code more straight forward as you then don't need to have the 'Schema' field and be incorporating it into every map function.
Also, you can (and should) have multiple different map/reduce pairs in a given design view. This is important because if you have two different document "Schema"s emit(doc.id, doc.val) how can you tell which is which for reduce purposes.
A more CouchDB idiomatic way to look at your data would be that you don't save the post_ids on the user. Just the UserID on the Posts, then have a map something like the following for Posts:
(doc) ->
emit([doc.user_id, doc.isFeatured], null);
emit([doc.isFeatured, doc.createdAt], doc.user_id);
Then a request to the view API with arguments like ?start_key=['12345345345234234']&end_key=['12345345345234234',{}] would get all their posts.
Where one with ?key=['12345345345234234', 1] would just get their featured posts.
The second emit also gives you ability to quickly get all of the posts that are featured across the whole system sorted by date -- with who made them if you want that data, without getting the whole of the posts sent down the pipe.