I am using sails.js with postgres as the database. Although most of the actions can be easily handled via waterline ORM, there are certain cases I prefer to use native queries and sometimes even postgres' native stored functions. However, the challenge with stored functions is that they come with an overhead of code maintenance.
In my project repo, I have created a directory sql which contains all the SQL functions. Currently, I have to manually make sure that whenever I am making some changes to a function, I need to recompile it on database.
I need to configure it such that these are compiled whenever I restart the server, just like all the models are re-created. Is it possible to do this and how ?
Sails doesn't have any built-in support for compiling Postgres stored procedures, but you could make a Grunt task for this. Take a look at the documentation for tasks. These are run every time you lift Sails (and in some cases, when files are changed).
A quick Google found the grunt-pg-utils package, which might help you on your way.
Related
I'm working on a sails app that I'll only use locally, so I'm using the sails-disk default for persistence. I would like to be able to backup and restore the data I put in there, though. Is this possible?
I didn't see anything in the waterline docs about working with the store other than via the API. I can write some sails code to export/import a dump, but if there's already something available or a standard way of doing this I'd prefer not to reinvent the wheel.
sails-disk, by default, stores data in .tmp/localDiskDb.db file inside your Sails project.
One way to take backup/restore is make copy of that file.
We are working on an application that will be offered both as a web-based and as a cross-platform desktop solution by means of Electron.
Due to customer requirements, the desktop client cannot make use of "the cloud" to store data; all data should be stored in the local machine or, even better, the user should have the option to keep the database/data file on an external HDD so that another user on the same local network can use the same data file.
We've been looking at NeDB, PouchDB, etc, but all these use either Web SQL or IndexedDB on the browser itself to store the data.
NeDB can theoretically use the file system but that seems only possible for Node Webkit apps.
Another option is of course MongoDB, but it requires setting up a site on a web server. Seeing as how our users will set that up in on their own machines, that will work for one user only but would make it very hard for them to share the data (note: assume users with little technical know-how).
Is there a way to force NeDB to persist data in a file instead of the in-browser database?
Alternatively, does any one know of a file-based, compact database that plays well with electron/node?
We'd preferably like to use a NoSQL database, but options of file-based SQL databases will be considered as well.
I have some experience with NeDB in an Electron app and I can say it will definitely work on the filesystem.
How are you initializing NeDB (or whatever your database choice is)? Also, are you initializing it in the main or renderer process? If you can share that, I think we could trace the issue to a configuration issue.
This is how you start NeDB with a persistent data-store that saves to disk.
var Datastore = require('nedb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
I think MongoDB is going to be overkill for an Electron app (it's meant to be really a high performance, distributed database running in the cloud).
Another option you could consider is LevelDB (a key/value store that can persist to the filesystem) which is popular in the node community. (EDIT 4/17/17 IndexedDB uses LevelDB underneath the hood, so if you go that route, may as well just use that)
One aspect I would definitely evaluate carefully is: How difficult is this database going to be to package and distribute? How do I integrate it into my build system? Level and NeDB can be included simply via npm install and any native code compiling is handled seamlessly with node-gyp, which is as simple as it gets. However, bundling Mongo, for example, will require some work to get a working build for each different platform.
On the side of my Gooddata project, I maintain a small PostgreSQL database that contains a few tables.
I would like to be able to integrate both my ETL processes using the same tool, and it seems to me cloudconnect would be the easiest way, since I already have my whole GoodData ETL in it.
Here are the ways I tried to do it without success:
I tried to have a look in the documentation, and it seems to me that all the functionalities of CloverETL that enabled this (DBOutput, PostGreSQLDataWriter) are not available in Cloudconnect.
I managed to connect to the Agile Datawarehouse Service (Database attached to GoodData), but it seems that only the ADS database is able to understand the request:
COPY MyDataBaseTable (field1,field2) FROM LOCAL '${DATA_TMP_DIR}/CIforADS.csv'
even when I adapt the syntax to PostgreSQL because the dynamic addressing I use here does not seem to work.
Is there any way to proceed that I'm missing? Can anyone think of a workaround?
In general this could be achieved by using of "DBExecute" component, but
I'm not sure if I understand it well - do you want to load data into your own Postgres instance using CloudConnect?
As a webapp novice, I'm not sure if I need to define models.py.
I already have a working Postgres database on Heroku that I've linked up with Postico and pgAdmin. Using these GUIs, it seems I can get and post data, and make structure changes very simply.
Most tutorials seem to glaze over the details and reasoning for having a models.py. Thanks!
Web frameworks typically enforce or encourage a Model-View-Controller (MVC) patterns that structures code such that the database code is kept separate to the presentation layer.
Frameworks like django come with and are more integrated with ORM functionality which is used to implement an MVC framework. The ORM allows you to programatically interact with your database without having to write sql code. It can let you create a schema as well as interact with it by mapping programming classes to tables and objects to rows.
Flask can be distinguished from many other web frameworks, like django, in that it is considered a micro framework. It is light weight and can be extended by adding extensions. If you need the database integration then you can use it with an external ORM tool like sqlalchemy (and optionally flask-sqlalchemy extension). You can then define a sqlalchemy model, for instance, in a file called model.py or schema.py, or any other name you find appropriate.
If you only need to run one or two queries against an existing postgres database and feel you have no need for the use of an ORM, you can simply use flask with the postgres driver and write the sql yourself. It is not mandatory to have a model.
A model/ORM can be beneficial. For example if you want to recreate an integration/test instance of the database, you can instruct the ORM tool to create a new instance of the database on another host by deploying the model. A model also provides a programming abstraction to the database, which in theory should make your code more database independent (well in theory, its hard to achieve this as databases can have subtle differences), and should make your code less tied to a specific database solution. Also, it alleviates the need of writing a language within a language (sql text strings within python code).
I have been working on a side project for the last few weeks, and built the system with EntityFramework Code first. This was very handy during development, as any changes i needed to make to the code were reflected in the DB nice and easily. But now that i want to launch the site, but continue development, i dont want to have to drop and recreate the DB every time i make a tweak to a model...
Is there a way to get EF to generate change scripts for the model change so i can deploy them myself to the production server? And how do i use the database somewhere else (Windows Service in the background of the site) without having to drop and recreate the table, and use the same model as I have already? Kind of like a "Code first, but now i have a production DB, dont break it..."
Personally i use the builtin data tools in VS2010 to do a database schema synchronization for updating production.
Another cheaper tool if you dont have VS Premium is SQLDelta which ive used in the past and is really good.
Both tools connect to the two database versions and allow you to synchronise the table schemas first. Both also have an export to SQL script functionality.
Comming up for EF is Migrations which allows you to solve just this problem within your solution however its still in beta. Migrations lets you describe upgrade and downgrade events for your database in code.
No RTM version of EF has this feature. Once you go to production you must handle it yourselves. The common way is to turn off database initializer in production and use some tool like VS Premium or RedGate Database compare to compare your production and dev database and create change SQL script.
You can also try to use EF Migrations which is exactly the tool you are asking for. The problem is it is still beta (but it should be part of EF 4.3 once completed) so it doesn't have to work in all cases and functionality / API can change in RTM.