Sometimes I need to run maintenance queries, like backfilling data of a new field I just created. How do I do this easily?
Firebase Functions seems targeted towards triggering on certain keywords or on certain schedules.
Basically, whatever I would do in MongoDB's terminal at will, I'd like to do in Firestore easily (even if I have to deploy code).
There is no set of "best practices" per se, but you can easily write programs to access Firebase and Google Cloud resources from your desktop. Just do some web searches for the server SDKs that provide access (there are many for different languages), and initialize them with a service account that has access to write data in the project.
Related
I am looking for a way to fetch data to my flutter app which can be adjusted and modified dynamically after deploying the app. As an example, if I want to change the images of the carousel depending on promotions or launch new books to the digital library. I need an economic option to host the data in cloud storage and fetch it from there.
I have considered firebase as well as google drive, but have yet to find a good guide. being a beginner and having concerns about security I want some expert advice if possible.
*edit-
Seeing many a tutorial I assume there is no better way than linking file URLs from the
Cloud Storage. So to dynamically change those is it possible to refer the URLs to some excel sheet fields to obtain URLs. Those fields can certainly be adjusted then without any hard coding. but the question is how to refer to such a sheet file? *
I also want to segregate the users into paid and free users, I have successfully proceeded with the authentication with firebase but still don't understand(I do have some concepts but don't know where to do that) how to put them in groups and impose limitations on them about accessing the data. any guidance, links and helpful advice will be cordially appreciated.
According to what you are looking for, I highly recommend you to use Firebase Remote Config, which is a cloud tool that allows you to modify your app's functionality and appearance without forcing users to download an update. You define in-app default values that control the functionality and appearance of your app when you use Remote Config. Then, for all app users or for subsets of your user base, you may utilize the Firebase console or the Remote Config backend APIs to modify in-app default values.
Your program can control when updates are applied, and it can check for updates regularly and apply them with minimal performance impact.
Remote Config comes with a client library that takes care of essential functions like fetching parameter values and caching them while still allowing you to manage when new values are active and how they affect the user experience in your app.
Here is a tutorial that uses Flutter and Firebase Remote Config that could also help you.
Background:
I'm building an API service app. The app is just like any other, you send an HTTP request and receive a response. This seems simple up until I start thinking about user registration, payments, authentication, logging and so on.
Application:
tl;dr simple app diagram
Endpoints listening for HTTP requests and doing all the request related work. This is the core of the service, what the service user would use this app for. Directly not accessible to the end user (unless somehow it knows the url). Python flask server, deployed on google cloud RUN.
API gateway acting like a proxy and a single access point forwarding the requests to the endpoints. This is the service access point for the end users. This part will also be responsible for authentication, limitations, logging and tracking the use of the API endpoints. Python flask server, deployed on google cloud RUN.
Website including documentation, demo and show off of API calls through API gateway, registration, payment (thinking of Stripe) etc. VueJS app on NodeJS server on google cloud compute VM.
Database storing credentials of registered users, payment information and auth keys. Not implemented yet.
Problems:
Is this architecture proper? What could be done differently or improved? How could I further simplify all the interactions between separate parts of the app? Am I not missing any essential parts?
Haven't yet implemented the database part and I'm not sure what should I
use? There are plenty of options on google cloud. Also I could go with something simple and just install a DB with http/JSON interface on google cloud compute VM. How do I chose the DB? Given such an app, what would be the best choice?
Please recommend literature/blogs/other sources of info on similar app
architecture for new developers not familiar with it?
This is pretty open ended, but here are some general comments:
Think about how your UI will work. Are you setting up a static app served directly from cloud storage or do you need something rendered on the server? Personally I prefer separating UI from API when I can but you need to be aware of things like search engine optimization. Even if you need to render some content dynamically your site can still be static. Take a look at static site generators like Gatsby. I haven't had to implement a server rendered UI in years and that makes me happy.
API gateway might be fine, but you don't really need it for anything. It might be simpler to start without it and concentrate on what actually matters. If your APIs are being called by an external client you can't trust the calls anyways and any API key you might be using will be exposed. I'd say don't worry about it for a single app. That being said, if you definitely want to use a GW then use one, just be aware that it is mostly a glorified proxy and not some core part of your architecture.
Make sure your API implementations don't store any local state so you can rely on Cloud Run scaling your services up and down. Definitely don't ever store state directly inside your containers. If you need state on the server it needs to be in some external data store.
Use JWTs or an external IDM (that will generate JWTs) for authentication. Keep session data on the client side as much as possible and pass the JWT in every API call to authenticate the caller. If you are implementing login on your own the only APIs you need to expose without tokens are for auth and password recovery, which you can separate into their own service.
Database selection depends on how well you understand your processes, how transactional your services are and your existing skillset. Overall I would use what you are comfortable with, you can probably succeed with a lot of things. Certain NoSQL flavors can seem simple on the surface but if you don't have a clear understanding on the types of queries you need to run they can get tedious to work with. Generally you should stick to relational databases for OLAP style implementations and consider NoSQL for OLTP. Personally I like MongoDB and it is very popular, probably because it sort of sits in the middle of the pack which makes it fit a lot of applications. Using MongoDB also makes you cloud agnostic since it is available on every platform. Using platform specific database flavors can lock you down to a specific vendor.
Whatever you do, don't start installing things on VMs. You can be almost 100% sure you are doing it wrong if this comes up. Remember, the services you consume don't all have to be managed by Google or even run on GCP. You can get MongoDB capacity directly from MongoDB who manage it on your behalf on all of the Big3 cloud vendors.
At least think about the long term, even if you don't necessarily need to have it impact your architecture right now. If you are expecting your app to be up for years try to make it more platform agnostic than less. This might mean sticking away from some really platform specific serverless features that will force you to jump a couple of extra hoops. If you are using Cloud Run you are using containers which already makes your app pretty portable, don't lock it to one platform by using a lot of platform specific features. That being said, don't stay away from them either. You should always go for the low hanging fruit, so don't try to avoid using things like secrets manager etc. If your app has a short lifespan and you need really fast time to market then don't worry about it.
Just my 2c, what you are doing is very generic and can be done in a lot of different ways.
THE PROBLEM
I'm writing a mobile app which will allow a user to log in, save some preferences that must be stored in a database, and display congressional bills to the user.
I've only written simple RESTful services with PHP and MySQL in the past. I'd like to take advantage of newer technologies, and am a little lost on general direction.
The bill data (formatted as JSON) can be gathered by running the scrapers found here. Using docker, I managed to set a working directory and download the files on my local machine.
I've designed a MySQL database for holding the relevant bill and user data.
I started to mess around in Google Cloud Platform, and read the doc that describes different models. I'm thinking of a few different ideas, but aren't familiar with GCP or what I can actually accomplish.
QUESTIONS
1) What are App Engine, Compute Engine, and Container Engine each for? I get the gist that Container Engine holds different instances of stuff you load up with docker, and that Compute Engine sets up a VM, but I don't really understand the relationships. How should I think of them?
2) When I run those scrapers from the shell, where are the files being stored, and how can I check on them? On my computer, I set a working directory, but how do directories work in GCP? Is it just a directory in the currently selected VM, or is this what Buckets are for?
IDEAS
1) Since my bill data already comes as JSON, should I skip the entire process of building a database for the bills and insert them into Firebase somehow? Is this even possible? If so, am I stuck using Firebase's NoSQL, or can I still set up a relational database?
2) I could schedule the scrapers to run periodically, detect new files, and run a script to parse the JSON and insert new bill data into my a database (PostgrSQL?/MySQL?). Then I would write an API.
3) Download the JSON files to a bucket, and write an API that reads from them. Not sure how the performance would compare to using a DB.
I'm open to other suggestions as well.
For your use case (stateless web application), App Engine is probably your best choice. The Google documentation has severalcomparisons of your computing options
You can use App Engine with PHP and cloud-hosted MySQL if you want, which could be a good way to get your toes wet without going in over your head.
I have a string for eg: "My name is John" stored in Firebase.
How would I query firebase so I can find all the posts in Firebase that have "John" ?
I can search for the first term in a string now using:
DataService.dataService.BASE_REF.child("Posts").child(selectedComment.commentKey).queryOrderedByChild("userComment").queryStartingAtValue(comment).queryEndingAtValue(comment+"\u{F8FF}").observeSingleEventOfType(.Value, withBlock: { (snapshot) in
where comment = "My"
I read about using Elastic search with Firebase but wanted to check if there was an easier way in Firebase before I looked at ElasticSearch/Flashlight for Firebase,
Unfortunately, Firebase doesn't support searching thru content like that (in any language SDK). From a Google Groups Post in July '16:
As a company that understands search, we're also a company that
understands using the best tool for the job. For fuzzy matching and
contains, a NoSQL, realtime data store isn't the correct tool--these
queries would be slow and scale poorly. BigQuery or ElasticSearch are
the right tool for providing useful results in a scalable and robust
manner.
Right now, this involves deploying a small node script to sync your
search results with the realtime data, as explained in the article
with the sample Flashlight lib. In the future, it will become more
"effortless" as we add integrations between Firebase and Cloud
products, particularly Cloud Functions and BigQuery interoperability.
BigQuery is, as I understand it, not specifically designed for user-facing search.
Elasticsearch (specifically, the Firebase plugin Flashlight) is a potential solution, but as you alluded to, it's an incredible amount of overhead (deploying/managing or renting an ES cluster, configuring the plugin, etc.). If content search is an important enough part of your app to justify that time/$, you may want to consider solutions beyond Firebase for your database needs, as it's by far one of the service's weakest areas.
In my opinion, you have a few options beyond Flashlight:
Algolia, a Search-as-a-service provider, does offer integration with Firebase, but I've never used it & so can't offer much more than to say that it exists.
Another alternative might be maintaining a collection of documents you want to search on another service, like AWS Cloud Search
Depending on the stage of your project & your needs, consider other Backends-as-a-Service that support more in terms of querying. E.g., GraphQL-as-a-service backends, like Scaphold.io, Graph.cool, and Reindex are all built on SQL databases, and (I believe) all support multiple types of querying.
In search of a 'Cloud Content Management System' like http://osmek.com/,
I could not find a single other CCMS that does what I want it to do :)
Basically, what I need is content management without a website frontend attached.
Just basic storage of data, documents, images, etc. etc. with a simple API to access, like Osmek. Just NoSQL or SQL based services won't do, because there can be images or documents attached. And, ofcourse, I'd like to have a backend to manage the data (like a typical CMS does) without writing a backend myself (if it's just the service)
Osmek is great, and it works most awesome in conjunction with Actionscript 3, but I'm just looking / searching for alternatives (if there even are any yet).
I need this form of hosted content management for content-manageing a mobile application.
So the question is: Is there anything else out there that does the same as osmek that you know of? OR, how do you manage application specific content?
Thanks!
I'd encourage you to take a look at Cloud CMS (http://www.cloudcms.com).
Cloud CMS is a JSON content management (CMS) platform built on top of MongoDB with a REST API and drivers for a variety of languages. You just drop in a driver and call methods to query, create, update and delete content.
The platform provides everything you need to power the back-end for mobile and HTML5 applications - from managing your content to managing users and groups, credentials, security tokens (OAuth2), Git-like collaborative workspaces, real-time analytics, activities, data transformations and more.
Everything runs in the cloud on an elastic back-end. It's probably more akin to Parse than a traditional CMS. You just make calls to the APIs. We keep the costs low by letting you only pay for what you use (almost like a utility). You just pay for storage and data transfer.
Disclaimer: I'm one of the founders of Cloud CMS. So I'm a pretty lousy reference in terms of its objective value. However, a couple of us worked at traditional "ECM" companies in the past and we think we've built something that puts a genuine beating on those guys.