Currently we are using apache flume framework to collect the logs from all our web applications and it will store the logs in MongoDB database table called “Raw_Data”.
Now we got a task to differentiate the logs and store the logging information in different tables based on the application. So we decided to maintain a separate table for each application. Is there any way to create and run stored JavaScript on a frequency like scheduler to load the data application wise in MongoDB?
I’m very much familiar with oracle DB but I’m new to MongoDb. Can someone help me to find out the required things for this asap.
Thanks
Kishore
MongoDB has stored Javascript procedures, but no scheduler.
http://docs.mongodb.org/manual/tutorial/store-javascript-function-on-server/
You could trigger this from a client machine using a cron job and the mongo shell. However, I strongly recommend simply running a script in Python or similar client-side scripting language. It'll be far easier to version-control and to debug than server-side Javascript.
Related
I use node-red to create an Api from a server. I want to read and send data via http. I use the browser-based programming method. I want to send data from a postgresql database. I installed the package node-red-contrib-postgres-multi. I don´t know how to put data into the database and how to read data from the database, because I cannot find examples.
Does anybody knows, how I can do that?
You can use postgrestor postgrestor package where you can define query or data modification statement directly in node window. It also allows parametrized queries via template engine.
The problem I've encountered is working with more than one db instance in the same flow but if you need only one db instance connection it should work for you.
Anyone know if we can configure a different database than the default
mongodb database that shipped with formio? I found "formio-sql" and it seems to act as a connector to our own database where we have to create and maintain own database tables and configure them with actions. What I am actually looking for is whether formio can use different databases like Mysql to store the submitted data natively.
As of today, it is not possible to use databases other than MongoDB to store submitted data natively.
The only option is by using 'Sql Connector' action via 'formio-sql' which you have seem to have found yourself.
we have a Ruby server application running on MongoDB.
We need to run function on a server which calculates sizes of objects per customer. We used db.command(:$eval...) for this however that's not deployable to production because such app would require rights to the whole db.
We were investigating to use db.system.js.save() . It is possible to run such a saved function withou using db.command(:$eval...) ?
Or would be other options for this? I'm currently also looking at the possibility to use mapReduce for this.
Thanks in advance,
Michal
THE PROBLEM
I'm writing a mobile app which will allow a user to log in, save some preferences that must be stored in a database, and display congressional bills to the user.
I've only written simple RESTful services with PHP and MySQL in the past. I'd like to take advantage of newer technologies, and am a little lost on general direction.
The bill data (formatted as JSON) can be gathered by running the scrapers found here. Using docker, I managed to set a working directory and download the files on my local machine.
I've designed a MySQL database for holding the relevant bill and user data.
I started to mess around in Google Cloud Platform, and read the doc that describes different models. I'm thinking of a few different ideas, but aren't familiar with GCP or what I can actually accomplish.
QUESTIONS
1) What are App Engine, Compute Engine, and Container Engine each for? I get the gist that Container Engine holds different instances of stuff you load up with docker, and that Compute Engine sets up a VM, but I don't really understand the relationships. How should I think of them?
2) When I run those scrapers from the shell, where are the files being stored, and how can I check on them? On my computer, I set a working directory, but how do directories work in GCP? Is it just a directory in the currently selected VM, or is this what Buckets are for?
IDEAS
1) Since my bill data already comes as JSON, should I skip the entire process of building a database for the bills and insert them into Firebase somehow? Is this even possible? If so, am I stuck using Firebase's NoSQL, or can I still set up a relational database?
2) I could schedule the scrapers to run periodically, detect new files, and run a script to parse the JSON and insert new bill data into my a database (PostgrSQL?/MySQL?). Then I would write an API.
3) Download the JSON files to a bucket, and write an API that reads from them. Not sure how the performance would compare to using a DB.
I'm open to other suggestions as well.
For your use case (stateless web application), App Engine is probably your best choice. The Google documentation has severalcomparisons of your computing options
You can use App Engine with PHP and cloud-hosted MySQL if you want, which could be a good way to get your toes wet without going in over your head.
I am using sails.js with postgres as the database. Although most of the actions can be easily handled via waterline ORM, there are certain cases I prefer to use native queries and sometimes even postgres' native stored functions. However, the challenge with stored functions is that they come with an overhead of code maintenance.
In my project repo, I have created a directory sql which contains all the SQL functions. Currently, I have to manually make sure that whenever I am making some changes to a function, I need to recompile it on database.
I need to configure it such that these are compiled whenever I restart the server, just like all the models are re-created. Is it possible to do this and how ?
Sails doesn't have any built-in support for compiling Postgres stored procedures, but you could make a Grunt task for this. Take a look at the documentation for tasks. These are run every time you lift Sails (and in some cases, when files are changed).
A quick Google found the grunt-pg-utils package, which might help you on your way.