The A query returns the port and network data from MongoDB Datasource and I want the result of it to be used in the B query. So how can I approach it?
I am new to cloud-ant, In my current assignment i want to search all distinct records based on fields x:
I have documents which have domain as attribute. I want all unique domains which are present in my db.Below is the example,
documentNo1-{"domain":"gmail.com"}
documentNo2-{"domain":"ymail.com"}
documentNo3-{"domain":"gmail.com"}
expected result is API should return only unique domain name, like below
[gmail.com,ymail.com]
I am not getting operators in cloud-ant which can achieve this, only solution i have is to retrieve it and create our own unique domain list.
Looking for any good approach/solution for above scenario.
You can use Cloudant Search to create a faceted index.
See https://console.bluemix.net/docs/services/Cloudant/api/search.html#faceting
This would allow you to essentially group documents by domain, creating the unique list you need.
There is a good video tutorial showing this technique:
https://www.youtube.com/watch?v=9er3XI150VM
I have a simple model where I have a CLIENT class. Nodes from CLIENT can have a relationship OWS_MONEY_TO to another CLIENT. That's it.
With this setup, I created 10 million CLIENT nodes and 50 million random relationships of OWS_MONEY_TO within random CLIENTS.
When I run this query:
MATCH
{class:CLIENT, as:A}-OWS_MONEY_TO->{class:CLIENT, as:B}
RETURN A.name as Payer, B.name as Receiver limit 10
I hit this error:
Query 'SELECT FROM CLIENT' returned a result set with more than 10000 records.
Check if you really need all these records, or reduce the resultset by using a
LIMIT to improve both performance and used RAM
I'm using a limit already, as you may see, and I haven't been able to figure out how to get any result from this query.
I see having mulitple where acts like AND, but how about OR?
You can't really do an OR query in Cloud Firestore.
As a workaround, you could run two separate queries and merge them together on the client, or add some custom field that would essentially perform the "OR" query for you on the database. (For an example of that latter one, if you know you're going to often run an "age > 65 OR age < 18" query on the database, you could create a specific age_high_or_low field that you would set to true if the age field were greater than 65 or less then 18.)
I have two tables in PostgreSQL:
urls (table with indexed pages, host is indexed column, 30 mln rows)
hosts (table with information about hosts, host is indexed column, 1mln rows)
One of the most frequent SELECT in my application is:
SELECT urls.*
FROM urls
JOIN hosts ON urls.host = hosts.host
WHERE urls.projects_id = ?
AND hosts.is_spam IS NULL
ORDER by urls.id DESC, LIMIT ?
In projects which have more than 100 000 rows in urls table the query executes very slow.
Since the tables has grown the query is execution slower and slower. I've read a lot about NoSQL databases (like MongoDB) which are designed to handle so big tables and i'am taking into consideration move my data to MongoDB. Everything would be easy, if i didn't have to check hosts table during selecting data from urls table. I've heard that MongoDB doesn't support joins, so my question is how to solve above problem? I could put information about host in urls collection, but the field hosts.is_spam could be updated by user and i would have to update the whole urls collection. I don't know it it is right solution.
I would be greatful for any advices.
If you don't use joins, then relational dbs can also work pretty fast. I think, this is the case where you need to denormalize for the sake of performance.
Option 1
Copy is_spam column to the urls table. When this value of the host changes, update all related urls. This is okay if you don't do it too often.
Option 2
I don't know your app, but I assume that the number of spam hosts is relatively small. In this case, you could put their ids to an in-memory store (memcached, redis, ...), query all the urls and filter out spam urls in the app. This way your pagination gets a little broken, but sometimes it's a viable option.
You are correct in that the problem is the join, but my guess is that it's just the wrong kind of join. As Frank H. mentioned, PostgreSQL should be able to process this type of query rather handily depending on the frequency of hosts.is_spam. You probably want to cluster the urls table on id to optimize the order by-limit phase. Since you only care about urls.* you can minimize disk io by creating a partial index on hosts.host where is_spam is not null to make it easy to get just the short list of hosts to avoid.
Try this:
select urls.*
from urls
left join hosts
on urls.host = hosts.host
and hosts.is_spam is not null
where urls.projects_id = ?
and hosts.host is null
Or this:
select *
from urls
where urls.projects_id = ?
and not exists (
select 1
from hosts
where hosts.host = urls.hosts
and hosts.is_spam is not null
)
This will allow PostgreSQL to use an anti-join to pull only urls which are not mapped to a known spammy host. The results may be different from your query if there are urls with a null or invalid host.
It is true that MongoDB doesn't support joins. In a case like this, I would structure my urls collection like this
urls : {
name,
some_other_property,
host
}
You can then fetch the host for a particular URL, and check the is_spam field for it in the hosts collection. Note that this will need to be done by the client querying the DB and cannot be done at the DB itself like you would with a JOIN.
Similar to the answer by #xbones, but with specific examples
Putting a host_id field in your urls documents is one way to go. It will require that you first pull a result of url documents, and then pull a result of spam hosts, then filter locally in your client code
Roughly:
var urls = db.urls.find({projects_id:'ID'}, {_id: 1, host_id: 1});
var hosts = db.hosts.find({is_spam: 1}, {_id: 1});
# psuedocode
ids_array = _id for _id in urls if host_id is not in hosts
urls = db.urls.find({_id: {$in: ids_array}});
Or:
var urls = db.urls.find({projects_id:'ID'});
var hosts = db.hosts.find({is_spam: 1}, {_id: 1});
# psuedocode
urls = url for url in urls if host_id is not in hosts
The first example assumes the result of the project_id query could be huge (and your url documents are bigger) and you only wanted to grab the smallest amount of data possible, then you filter locally, and then batch get the full final url documents.
The second example just gets the full url documents to start, and filters them down locally.