I'm trying to run the upsert/delete some of the values in DB2 database source table, which is a existing table on DB2. Is it possible using Pyspark/Spark SQL/Dataframes.
There is no direct way for update/delete in relational database using Pyspark job, but there are workarounds.
(1) You can create a identical empty table (secondary table) in relational database and insert data into secondary table using pyspark job, and write a DML trigger that would perform desired DML operation on your primary table.
(2) You can create a dataframe (eg. a) in spark that would be copy of your existing relational table and merge existing table dataframe with current dataframe(eg. b) and create a new dataframe(eg. c) that would be having latest changes. Now truncate the relational database table and reload with spark latest changes dataframe(c).
These is just a workaround and not a optimal solution for huge amount of data.
I'm creating an application where the user can create tables in database (by using an assistant).
Sometimes I get a deadlock because I can't modify database structure while sessions are active (AccessExclusiveLock).
I'm trying to find a way to create a table without executing DDL script.
( or to figure out my current problem )
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
How can I link a Google spreadsheet to PostgreSQL? I googled and got some sample code for MySQL, and a couple of products which do it. As per this link ,support exists for a few databases.
My approach is using R and its googlesheets and DBI libraries. Googlesheets connects the spreadsheet with R and DBI connects the R with PostgreSQL. You can update the table whenever you want by simply running the script. With this approach you can also add transformations to the data before storing them in PostgreSQL. Another way is using Python and pandas and gspread libraries.
More info:
Googlesheets: https://cran.r-project.org/web/packages/googlesheets/vignettes/basic-usage.html
DBI: https://cran.r-project.org/web/packages/DBI/vignettes/DBI-1.html
We have been pulling Google Sheets into QGIS via PostgreSQL Foreign Data Wrappers. We then build a materialized view that connects records to geometry (via School Numbers, for example) and use the materialized view as a standard spatial table.
From there we have also published this 'google sheet materialized view' to a web application via node.js and add a button to refresh the map if the google sheet data has been changed. It works really well.
Method 1: Multicorn FDW and GSheets Extension:
(Note: The gsspreadsheet extension to the multicorn FDW is questionably maintained, and there is the risk that v4 of the GSheets API may break it... we don't know yet, and are planning for the possibility that we might have to implement Method 2 (below) if it does indeed break. )
To connect to a Google Sheet via a PostgreSQL FDW, use the Multicorn FDW:
https://github.com/Kozea/Multicorn
Then install the gspreadsheet_fdw extension for Multicorn:
https://github.com/lincolnturner/gspreadsheet_fdw
In PostgreSQL, create the multicorn FDW:
CREATE SERVER multicorn_gspreadsheet
FOREIGN DATA WRAPPER multicorn
OPTIONS (wrapper 'gspreadsheet_fdw.GspreadsheetFdw');
Then create the FDW table connecting to the Google Sheet using the gspreadsheet extension of multicorn:
CREATE FOREIGN TABLE speced_fdw.centerprogram_current_gsheet (
column1 integer NULL,
column2 varchar NULL,
column3 varchar NULL
)
SERVER multicorn_gspreadsheet
OPTIONS (keyfile '/usr/pgsql-9.5/share/credential.json', gskey 'example_d33lk2kdislids');
You now have a foreign table pointing directly to your Google Sheet, which you can built a materialized view from.
Method 2: Using FILE_FDW and CSV GSheet Export:
The other option is to use the FILE_FDW right out of PostgreSQL and connect to the CSV export of the GSheet using WGET.
First, create the server:
CREATE SERVER fdw_files
FOREIGN DATA WRAPPER file_fdw
OPTIONS ()
Then create the FDW table:
CREATE FOREIGN TABLE public.test_file_fdw (
"name" varchar NOT NULL,
"date" varchar NULL,
"address" varchar NULL
)
SERVER fdw_files
OPTIONS (program 'wget -q -O - "https://docs.google.com/spreadsheets/d/2343randomurlcharacters_r0/export?gid=969976&format=csv"', format 'csv', header 'true');
The above options for wget are listed here: https://www.gnu.org/software/wget/manual/html_node/HTTP-Options.html
You can do this using Google app script and database connector like psycopg2 if you're using python or node-postgres if you're using node. Run your db connector on some server, front it with a REST API and fetch the data using app script from google sheet.
One thing you'd need to figure out is permissions (Google calls them scopes).
You can build an addon and publish it internally. Here is a good tutorial on how to do this.
If you don't want to build your own addon, you can use database connectors like Castodia, Kpibees and Seekwell from Google marketplace. Most of them support Postgres.
Wow, I forgot about this question. I got this working by making an additional MySQL database for two databases. The Mysql database had a single table which was used to talk to the google sheets api. In turn this mysql table was a foreign table in the Postgres database.
I am developing a windows application and using Postgres as backend database. At some point in my application i am dynamically creating table e.g Table1, then Table2 and so on. In this way i have many dynamic table in my database. Now i provide a button "Clean Database", so i need to remove all those dynamic tables using SQL query. Should some one guide me how to write SQL Query that automatically delete all such tables?
You should just be able to say
DROP TABLE {tablename}
for each dynamically created table. Try that and see if it works.
I have a small database with a table with 2 columns (ID and Name)
I have an ASP.Net form that i want to input an ID and a Name and upon clicking the button, it will insert rows into the database
I want to do this using a dataset. Meaning, I want to add the rows first into a ado.net dataset. Once I have the dataset with its table and rows, I want to insert data into the SQL Server 2008 database using SQLDataAdpater.
I can manually create a DS and add rows in it
How do I accomplish inserting data to database with the dataset I created?
Maybe this will help you
HOW TO: Implement a DataSet SELECT INTO Helper Class in Visual C# .NET
http://support.microsoft.com/kb/326009/en