I have a string for eg: "My name is John" stored in Firebase.
How would I query firebase so I can find all the posts in Firebase that have "John" ?
I can search for the first term in a string now using:
DataService.dataService.BASE_REF.child("Posts").child(selectedComment.commentKey).queryOrderedByChild("userComment").queryStartingAtValue(comment).queryEndingAtValue(comment+"\u{F8FF}").observeSingleEventOfType(.Value, withBlock: { (snapshot) in
where comment = "My"
I read about using Elastic search with Firebase but wanted to check if there was an easier way in Firebase before I looked at ElasticSearch/Flashlight for Firebase,
Unfortunately, Firebase doesn't support searching thru content like that (in any language SDK). From a Google Groups Post in July '16:
As a company that understands search, we're also a company that
understands using the best tool for the job. For fuzzy matching and
contains, a NoSQL, realtime data store isn't the correct tool--these
queries would be slow and scale poorly. BigQuery or ElasticSearch are
the right tool for providing useful results in a scalable and robust
manner.
Right now, this involves deploying a small node script to sync your
search results with the realtime data, as explained in the article
with the sample Flashlight lib. In the future, it will become more
"effortless" as we add integrations between Firebase and Cloud
products, particularly Cloud Functions and BigQuery interoperability.
BigQuery is, as I understand it, not specifically designed for user-facing search.
Elasticsearch (specifically, the Firebase plugin Flashlight) is a potential solution, but as you alluded to, it's an incredible amount of overhead (deploying/managing or renting an ES cluster, configuring the plugin, etc.). If content search is an important enough part of your app to justify that time/$, you may want to consider solutions beyond Firebase for your database needs, as it's by far one of the service's weakest areas.
In my opinion, you have a few options beyond Flashlight:
Algolia, a Search-as-a-service provider, does offer integration with Firebase, but I've never used it & so can't offer much more than to say that it exists.
Another alternative might be maintaining a collection of documents you want to search on another service, like AWS Cloud Search
Depending on the stage of your project & your needs, consider other Backends-as-a-Service that support more in terms of querying. E.g., GraphQL-as-a-service backends, like Scaphold.io, Graph.cool, and Reindex are all built on SQL databases, and (I believe) all support multiple types of querying.
Related
The problem I'm dealing with is performance issues when needing to perform multiple Firestore queries in order to reach a result similar to to joins in SQL.
In my Flutter app, which uses Firebase and Firestore, I use Cloud Functions for serving my REST API. My API is the only thing that actually interacts with the Firestore database, not the Flutter app itself. I was always taught to design apps like this (not allowing the client to interact directly with the database) and so continued with that approach.
A potential solution I've seen are the official Firestore plugins for Flutter and have seen people designing their clients to interact directly with Firebase, however I haven't found any official documentation explicitly saying this design is preferred or even ok, and since it goes against when have been taught I haven't tried it yet. Can anyone confirm this or point me to documentation confirming this?
Edit
Perhaps it's worth noting that the reasons I've been told not to have the client connect directly to the database are 1. Security and 2. It would require business logic for managing data to be in the client and that should be handled server-side.
Sometimes I need to run maintenance queries, like backfilling data of a new field I just created. How do I do this easily?
Firebase Functions seems targeted towards triggering on certain keywords or on certain schedules.
Basically, whatever I would do in MongoDB's terminal at will, I'd like to do in Firestore easily (even if I have to deploy code).
There is no set of "best practices" per se, but you can easily write programs to access Firebase and Google Cloud resources from your desktop. Just do some web searches for the server SDKs that provide access (there are many for different languages), and initialize them with a service account that has access to write data in the project.
I am trying to shift towards serverless architecture when it comes to building REST API. I came from Ruby on Rails background.
I have successfully understood and adapted services such as Api Gateway, Cognito, RDS and Lambda functions, however I am struggling with putting it all together in optimal way.
My case is the following. I have a simple user based platform when there are multiple resources related to application members say blog application.
I have used Cognito for the sake of authentication and Aurora as the database service for keeping thing like articles and likes..
Since the database and Cognito user pool are decoupled, it is hard for me to do things like:
Fetching users that liked particular article
Fetching users comments
It seems problematic for me because I need to pass some unique Cognito user identifier (retrieved during authorization phase in API gateway) to lambda function which will then save the database record with an external reference to this user. On the other hand, If I were to fetch particular users, firstly I must fetch their identifiers from my relation database and then request users details from Cognito user pool..I lack some standard ways of accessing current user in my lambda functions as well as mechanisms for easily associating databse record with that user..
I have not found some convincing recommended patterns for designing such applications even though it seems like a very common problem and I am having hard time struggling if my approach is correct..
I would appreciate some comments on what are some patterns to consider when designing simple user based platform and what are the pitfalls of my solution. Any articles and examples will also be very helpfull.
Thanks in advance.
These sound like standard problems associated with distributed, indpependent, databases. You can no longer delegate all relationships to the database and get a result aggregating them in some way. You have to do the work yourself by calling one database, then the other.
For a case like this:
Fetching users that liked particular article
You would look up the "likes" database to determine user IDs of those who liked it, then look up the "users" database to determine user details such as name and avatar.
Most patterns follow standard database advice, e.g. in the above example, you could follow the performance-oriented pattern of de-normalising - store user data such as name and avatar against each "like", as long as you feel the extra storage and burden of keeping it consistent is justified by the reduction in queries (probably too many Likes to justify this).
Another important practice is using bulk queries to avoid N+1 queries. This is what Rails does with the includes syntax, but you may have to do it yourself here. In my example, it should only take two queries because the second query should get all required user data in one go, by querying for users matching the list of user IDs.
Finally, I'd suggest you try to abstract things. This kind of code gets messy fast, so be sure to build a well-encapsulated data layer that isolates application code from dealing with the mess of multiple databases.
I have to start an application where I need to decide which db should I use
For complex queries which should I prefer mongodb or firestore
Thanks
Personally, I would choose MongoDB but that strongly depends upon which application you want to build.
MongoDB is open source and not Cloud based, it is high portable and supports many programming languages and external products, although query syntax looks a bit more complicated.
If you need auto scaling feature and integration with other Google services then Firestore would be the best choice.
To wrap up, MongoDB seems to be more secure and consistent while Firestore is better suited for mobile apps or application based on real time data.
Does anyone have experiences with programmatic exports of data in conjunction with BaaS providers like e.g. parse.com or StackMob?
I am aware that both providers (as far as I can tell from the marketing talk) offer a REST API which will allow for queries against the database, not only to be used by mobile clients but also by e.g. custom web apps.
I am also aware that both providers offer a manual export of data (parse.com via their web interface, StackMob via support).
But lets say I would like to dump all data nightly, so that I can import it into a reporting system for instance. Or maybe simply to have an up-to-date backup.
In this case, I would need a programmatic way to export/replicate the data stored in the backend. Manual exports are not an option for obvious reasons.
The REST APIs offered however seem to be designed for specific queries, not for mass reads (performance?). Let alone the pricing - I assume none of the providers would be happy about a nightly X Gigabyte data export via their REST API, so their probably will be a price tag.
I just couldn't find any specific information on this topic so far, so I was wondering if anyone else has already gone through this. Also, any suggestions on StackMob/parse alternatives are welcome, especially if related to the data export topic.
Cheers, Alex
Did you see the section of the Parse REST API on Batch operations? Batch operations reduce the number of API calls needed to grab data so that you are not using a call for every row you retrieve. Keep in mind that there is still a limit (the default is 100, but you can set it to a maximum of 1000). That means you are still limited to pulling down 1000 rows per API call.
I can't comment on StackMob because I haven't used it. At my present job, we are using Parse and we wrote a C# app which compares the data in a Parse class with a SQL table and pulls down any changes.