I want to copy tables from a postgreSQL schema to Snowflake (including data). What's the easiest way to do this?
My postgreSQL database lives in AWS RDS.
I went through this a couple of months ago and will share what I learned.
Snowflake does not recognize psql so a psql export/import will not work.
The recommendation I received from support was to export the tables as csv files, recreate the DDL, and then stage the csv files in S3.
There are third party tools that help connect and sync Postgres databases to Snowflake. Stitch being the one my group looked at.
So you can write your own integration or use Stitch to connect the two. There could be a more effective way to do this but if you are moving a large database over, I am afraid there is not an easy way to do this. That said, it is not terribly difficult but will take time to move everything over. Hope that helps!
Please check out the following community article which explains the steps to migrate.
https://community.snowflake.com/s/article/PostgreSQL-to-Snowflake-ETL-Steps-to-Migrate-Data
This is from one of the Snowflake partners who provides the tools to migrate the data.
Related
I'm tryig to config multimaster replication between separated postgresql db's. The best non-commercial solution I found is SymmetricDS, but I can't figure out how to config it. Its documentation says alot about abstract concepts, but after reading it I still don't understand what I actually have to do. Maybe there are someone who had had the same probleme and could tell me in general steps how I can make this thing work?
Download the latest version. Then follow the step-by-step quick guide from the first chapter http://www.symmetricds.org/doc/3.10/html/tutorials.html#_demo on how to establish a two node replication. When it started working add a column to one of the tables, recreate the triggers and see how it gets synced. Then create a new table, declare a new trigger for it, associate it to an existing router, restart and enjoy it gets syncing.
On the side of my Gooddata project, I maintain a small PostgreSQL database that contains a few tables.
I would like to be able to integrate both my ETL processes using the same tool, and it seems to me cloudconnect would be the easiest way, since I already have my whole GoodData ETL in it.
Here are the ways I tried to do it without success:
I tried to have a look in the documentation, and it seems to me that all the functionalities of CloverETL that enabled this (DBOutput, PostGreSQLDataWriter) are not available in Cloudconnect.
I managed to connect to the Agile Datawarehouse Service (Database attached to GoodData), but it seems that only the ADS database is able to understand the request:
COPY MyDataBaseTable (field1,field2) FROM LOCAL '${DATA_TMP_DIR}/CIforADS.csv'
even when I adapt the syntax to PostgreSQL because the dynamic addressing I use here does not seem to work.
Is there any way to proceed that I'm missing? Can anyone think of a workaround?
In general this could be achieved by using of "DBExecute" component, but
I'm not sure if I understand it well - do you want to load data into your own Postgres instance using CloudConnect?
I'm wondering whether it's possible to make GAE create more than 7 sql database backups and how much that will cost. They don't seem to mention this possibility in their document.
I'm afraid it's not possible to have more than 7 backups at a time for Cloud SQL. The goal is to use the backups as a last line of defense for restoring a broken database. If you need snapshots over time, you can use the Export functionality to save dumps to a Cloud Storage bucket.
I'd like to use a Google spreadsheet to display my database analytics
I'd like to be able to do summary queries on my Heroku Postgres database using Google Apps Script and then display and chart them in a Google spreadsheet.
Heroku offers a number of ways to connect to Heroku Postgres:
https://devcenter.heroku.com/articles/heroku-postgresql
Likewise Google Apps script offers access to a number of different external services
https://developers.google.com/apps-script/defaultservices
I've never attempted this before and so am interested in what is simplest.
JDBC seems possible but are there any other options?
As far as I can see, the only overlap between the two is JDBC which I have no experience with but feels like a bit of a heavyweight third protocol to use to get between the systems.
IS JDBC the best way to get the data across or is there something simpler I'm missing?
Set up a dataclip from dataclips.heroku.com with your desired data described as a SQL query.
Append .csv to the resulting URL
Use that URL on the google spreadsheet's importData function, like so:
=importData("https://dataclips.heroku.com/[your-dataclip].csv")
I prefer to use Skyvia for connecting Google Sheets and Heroku Postgres without coding. Here is how I do it: https://skyvia.com/data-integration/integrate-google-sheets-heroku-postgres. All I need is to specify the connections to Google Sheets and Heroku Postgres and select data to replicate. Skyvia will copy the specified Google Sheets data to Heroku Postgres and maintain this copy up-to-date automatically with incremental updates.
QueryClips is exactly what you need. This is its primary use case.
I have been working with Microsoft SQL Server since 6.5 along with other database like Oracle, MySQL and SQLite. I equally appreciate or hate all these DBMS for some point or the other.
On our forthcoming project, we are considering Postgres in the back-end. I have already started playing with it, pretty interesting for me.
I have always heard good comments on Postgres database, but I don't like the admin studio at all. While creating new a table, I hate the way of creating columns on pgAdmin by having to click add button again and again.
Are there any "Studios" for Postgres database that provide
more organized table creation process (spreadsheet like)
graphical view designer
What's wrong with plain SQL? Writing plain SQL goes much faster than click-wait-click-wait-type-click-wait-ok-wait. You could use any tool for this, pgAdmin as well.
Open Office Base can also connect with PostgreSQL, works like MS Access. And talking about MS Access, MS Access can also connect to PostgreSQL to create tables, views, etc.
See this official list of Administration/Development tools.
EMS SQL Manager seems to be a good option.
for the graphical view of SQL queries
maybe SQLeo can Help