it my first time using google cloud.
I am setting up everything and I was going to add a sample database to my google cloud account to test few things, however when I try to import my sampleDB I get this error:
Could not complete the operation.
I have already make a bucket and imported my sql file in there,
this is my sample sql file:
> CREATE DATABASE IF NOT EXISTS student DEFAULT CHARACTER SET utf8mb4
> COLLATE utf8mb4_0900_ai_ci; USE student;
>
> CREATE TABLE person ( id int(11) NOT NULL, name varchar(25) NOT
> NULL, age int(3) NOT NULL, sex text, email text NOT NULL,
> study varchar(20) NOT NULL, birthday date NOT NULL ) ENGINE=InnoDB
> DEFAULT CHARSET=utf8mb4;
>
> INSERT INTO person (id, `name`, age, sex, email, study, birthday)
> VALUES (1, 'Saeed', 30, 'M', 'nakafisarrd#gmail.com', 'computer',
> '1987-04-30');
>
> ALTER TABLE person ADD PRIMARY KEY (id);
this is the tutorial I am following.
further more I have installed google app engine sdk for node. is doesn show me any error so I cant figure out what is going wrong here!
I think you might be mixing two topics here, so I will guide you through both process:
· First, you want to create a Cloud SQL instance importing a *.sql file, as explained in the tutorial you followed. However, bear in mind that this tutorial is using MySQL, so you should be creating a PostgreSQL instance instead if that fits your environment better. I would also recommend you to follow the official documentation on that topic, as it is explained in a clear way how to achieve what you want to do. I have tested the script you provided in a 2nd Generation MySQL instance changing the first line from CREATE DATABASE IF NOT EXISTS student DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci; to CREATE DATABASE IF NOT EXISTS student; and it worked.
· Second, you want to upload your NodeJS application to the Google Cloud Platform, and with that purpose I suggest using App Engine. As of today, NodeJS support is only available in App Engine Flexible, so you can also follow a guided quickstart in the documentation.
So I would recommend you to first clarify if you want to use either MySQL or PostgreSQL, create a Cloud SQL instance that matches your requirements, and then import the .sql file you shared. Everything should work fine.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
How can I link a Google spreadsheet to PostgreSQL? I googled and got some sample code for MySQL, and a couple of products which do it. As per this link ,support exists for a few databases.
My approach is using R and its googlesheets and DBI libraries. Googlesheets connects the spreadsheet with R and DBI connects the R with PostgreSQL. You can update the table whenever you want by simply running the script. With this approach you can also add transformations to the data before storing them in PostgreSQL. Another way is using Python and pandas and gspread libraries.
More info:
Googlesheets: https://cran.r-project.org/web/packages/googlesheets/vignettes/basic-usage.html
DBI: https://cran.r-project.org/web/packages/DBI/vignettes/DBI-1.html
We have been pulling Google Sheets into QGIS via PostgreSQL Foreign Data Wrappers. We then build a materialized view that connects records to geometry (via School Numbers, for example) and use the materialized view as a standard spatial table.
From there we have also published this 'google sheet materialized view' to a web application via node.js and add a button to refresh the map if the google sheet data has been changed. It works really well.
Method 1: Multicorn FDW and GSheets Extension:
(Note: The gsspreadsheet extension to the multicorn FDW is questionably maintained, and there is the risk that v4 of the GSheets API may break it... we don't know yet, and are planning for the possibility that we might have to implement Method 2 (below) if it does indeed break. )
To connect to a Google Sheet via a PostgreSQL FDW, use the Multicorn FDW:
https://github.com/Kozea/Multicorn
Then install the gspreadsheet_fdw extension for Multicorn:
https://github.com/lincolnturner/gspreadsheet_fdw
In PostgreSQL, create the multicorn FDW:
CREATE SERVER multicorn_gspreadsheet
FOREIGN DATA WRAPPER multicorn
OPTIONS (wrapper 'gspreadsheet_fdw.GspreadsheetFdw');
Then create the FDW table connecting to the Google Sheet using the gspreadsheet extension of multicorn:
CREATE FOREIGN TABLE speced_fdw.centerprogram_current_gsheet (
column1 integer NULL,
column2 varchar NULL,
column3 varchar NULL
)
SERVER multicorn_gspreadsheet
OPTIONS (keyfile '/usr/pgsql-9.5/share/credential.json', gskey 'example_d33lk2kdislids');
You now have a foreign table pointing directly to your Google Sheet, which you can built a materialized view from.
Method 2: Using FILE_FDW and CSV GSheet Export:
The other option is to use the FILE_FDW right out of PostgreSQL and connect to the CSV export of the GSheet using WGET.
First, create the server:
CREATE SERVER fdw_files
FOREIGN DATA WRAPPER file_fdw
OPTIONS ()
Then create the FDW table:
CREATE FOREIGN TABLE public.test_file_fdw (
"name" varchar NOT NULL,
"date" varchar NULL,
"address" varchar NULL
)
SERVER fdw_files
OPTIONS (program 'wget -q -O - "https://docs.google.com/spreadsheets/d/2343randomurlcharacters_r0/export?gid=969976&format=csv"', format 'csv', header 'true');
The above options for wget are listed here: https://www.gnu.org/software/wget/manual/html_node/HTTP-Options.html
You can do this using Google app script and database connector like psycopg2 if you're using python or node-postgres if you're using node. Run your db connector on some server, front it with a REST API and fetch the data using app script from google sheet.
One thing you'd need to figure out is permissions (Google calls them scopes).
You can build an addon and publish it internally. Here is a good tutorial on how to do this.
If you don't want to build your own addon, you can use database connectors like Castodia, Kpibees and Seekwell from Google marketplace. Most of them support Postgres.
Wow, I forgot about this question. I got this working by making an additional MySQL database for two databases. The Mysql database had a single table which was used to talk to the google sheets api. In turn this mysql table was a foreign table in the Postgres database.
Is there a free web based tool (I prefer Microsoft because I have an MSDN account) to view the database schema and its properties? I'm currently using Visio but I dont know if it's possible in Visio to do the following:
View the database schema that I will upload in a Sharepoint workspace.
If I click the name of the Database table, it will either pop up a small window with its database properties and column description or another window will pop up with its database properties and column description.
For example I have a Schema table interface composed of a profile and time tables.
THESE ARE JUST EXAMPLES.
ProfileDim table
Profile ID number(10)
FirstName varchar (20)
MiddleName varchar (20)
LastName varchar (20) TimeDim Table
DateHired datetime---------------EmployeeStartDate datetime,
Date_Name nvarchar(50),
Year datetime,
If I click the name "TimeDim" it will take me to another page where I can see the database properties written below and the description of each column. I prefer not to use MS excel or word. I want a tool that I could edit, delete in case I want to add another column, change the data type and change the name etc.
TimeDim table
COLUMN NAME DATA TYPE ALLOW NULLS DESCRIPTION
EmployeeStartDate datetime, Unchecked "START DATE OF THE EMPLOYEE"
Date_Name nvarchar(50), Checked "CURRENT DATE NAME"
Year datetime, Checked "NAME OF THE CURRENT YEAR"
There are several tools that generate your database documentation to for example html. I don't know online tools but the generated docs can be placed online ofcourse. They are not free but you can use them in the evaluation perio to generate the database documentation
toad data modeler
dbdesc
dezign for databases
For Microsoft DB servers, you can use SQL Server Management Studio Express.
The DB schema management tool depends on which database software you are using. There are different tools to MySQL, PostgreSQL, Microsoft SQL Server, etc...
My question is rather simple. I'm aware of the concept of a UUID and I want to generate one to refer to each 'item' from a 'store' in my DB with. Seems reasonable right?
The problem is the following line returns an error:
honeydb=# insert into items values(
uuid_generate_v4(), 54.321, 31, 'desc 1', 31.94);
ERROR: function uuid_generate_v4() does not exist
LINE 2: uuid_generate_v4(), 54.321, 31, 'desc 1', 31.94);
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
I've read the page at: http://www.postgresql.org/docs/current/static/uuid-ossp.html
I'm running Postgres 8.4 on Ubuntu 10.04 x64.
uuid-ossp is a contrib module, so it isn't loaded into the server by default. You must load it into your database to use it.
For modern PostgreSQL versions (9.1 and newer) that's easy:
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
but for 9.0 and below you must instead run the SQL script to load the extension. See the documentation for contrib modules in 8.4.
For Pg 9.1 and newer instead read the current contrib docs and CREATE EXTENSION. These features do not exist in 9.0 or older versions, like your 8.4.
If you're using a packaged version of PostgreSQL you might need to install a separate package containing the contrib modules and extensions. Search your package manager database for 'postgres' and 'contrib'.
Without extensions (cheat)
If you need a valid v4 UUID
SELECT uuid_in(overlay(overlay(md5(random()::text || ':' || random()::text) placing '4' from 13) placing to_hex(floor(random()*(11-8+1) + 8)::int)::text from 17)::cstring);
Thanks to #Denis Stafichuk #Karsten and #autronix
Or you can simply get UUID-like value by doing this (if you don't care about the validity):
SELECT uuid_in(md5(random()::text || random()::text)::cstring);
output>> c2d29867-3d0b-d497-9191-18a9d8ee7830
(works at least in 8.4)
PostgreSQL 13 supports natively gen_random_uuid ():
PostgreSQL includes one function to generate a UUID:
gen_random_uuid () → uuid
This function returns a version 4 (random) UUID. This is the most commonly used type of UUID and is appropriate for most applications.
db<>fiddle demo
The answer by Craig Ringer is correct. Here's a little more info for Postgres 9.1 and later…
Is Extension Available?
You can only install an extension if it has already been built for your Postgres installation (your cluster in Postgres lingo). For example, I found the uuid-ossp extension included as part of the installer for Mac OS X kindly provided by EnterpriseDB.com. Any of a few dozen extensions may be available.
To see if the uuid-ossp extension is available in your Postgres cluster, run this SQL to query the pg_available_extensions system catalog:
SELECT * FROM pg_available_extensions;
Install Extension
To install that UUID-related extension, use the CREATE EXTENSION command as seen in this this SQL:
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
Beware: I found the QUOTATION MARK characters around extension name to be required, despite documentation to the contrary.
The SQL standards committee or Postgres team chose an odd name for that command. To my mind, they should have chosen something like "INSTALL EXTENSION" or "USE EXTENSION".
Verify Installation
You can verify the extension was successfully installed in the desired database by running this SQL to query the pg_extension system catalog:
SELECT * FROM pg_extension;
UUID as default value
For more info, see the Question: Default value for UUID column in Postgres
The Old Way
The information above uses the new Extensions feature added to Postgres 9.1. In previous versions, we had to find and run a script in a .sql file. The Extensions feature was added to make installation easier, trading a bit more work for the creator of an extension for less work on the part of the user/consumer of the extension. See my blog post for more discussion.
Types of UUIDs
By the way, the code in the Question calls the function uuid_generate_v4(). This generates a type known as Version 4 where nearly all of the 128 bits are randomly generated. While this is fine for limited use on smaller set of rows, if you want to virtually eliminate any possibility of collision, use another "version" of UUID.
For example, the original Version 1 combines the MAC address of the host computer with the current date-time and an arbitrary number, the chance of collisions is practically nil.
For more discussion, see my Answer on related Question.
pgcrypto Extension
As of Postgres 9.4, the pgcrypto module includes the gen_random_uuid() function. This function generates one of the random-number based Version 4 type of UUID.
Get contrib modules, if not already available.
sudo apt-get install postgresql-contrib-9.4
Use pgcrypto module.
CREATE EXTENSION "pgcrypto";
The gen_random_uuid() function should now available;
Example usage.
INSERT INTO items VALUES( gen_random_uuid(), 54.321, 31, 'desc 1', 31.94 ) ;
Quote from Postgres doc on uuid-ossp module.
Note: If you only need randomly-generated (version 4) UUIDs, consider using the gen_random_uuid() function from the pgcrypto module instead.
Update from 2021,
There is no need for a fancy trick to auto generate uuid on insert statement.
Just do one thing:
Set default value of DEFAULT gen_random_uuid () to your uuid column.
That is all.
Say, you have a table like this:
CREATE TABLE table_name (
unique_id UUID DEFAULT gen_random_uuid (),
first_name VARCHAR NOT NULL,
last_name VARCHAR NOT NULL,
email VARCHAR NOT NULL,
phone VARCHAR,
PRIMARY KEY (unique_id)
);
Now you need NOT to do anything to auto insert uuid values to unique_id column. Because you already defined a default value for it. You can simply focus on inserting onto other columns, and postgresql takes care of your unique_id. Here is a sample insert statement:
INSERT INTO table_name (first_name, last_name, email, phone)
VALUES (
'Beki',
'Otaev',
'beki#bekhruz.com',
'123-456-123'
)
Notice there is no inserting into unique_id as it is already taken care of.
About other extensions like uuid-ossp, you can bring them on if you are not satisfied with postgres's standard gen_random_uuid () function. Most of the time, you should be fine without them on
ALTER TABLE table_name ALTER COLUMN id SET DEFAULT uuid_in((md5((random())::text))::cstring);
After reading #ZuzEL's answer, i used the above code as the default value of the column id and it's working fine.
The uuid-ossp module provides functions to generate universally unique identifiers (UUIDs)
uuid_generate_v1() This function generates a version 1 UUID.
Add Extension
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
Verify Extension
SELECT * FROM pg_extension;
Run Query
INSERT INTO table_name(id, column1, column2 , column3, ...) VALUES
(uuid_generate_v1(), value1, value2, value3...);
Verify table data
SELECT uuid_generate_v5(uuid_ns_url (), 'test');
I have executed the following create statement using SQLWorkbench at my target postgresql database:
CREATE TABLE Config (
id serial PRIMARY KEY,
pub_ip_range_low varchar(100),
pub_ip_range_high varchar(100)
);
Right after table creation I request the table content by typing 'select * from config;' and see that table could be retrieved. Nevertheless, my java program that uses JDBC type 4 driver cannot access the table when I issue the same select statement in it. An exception is thrown when the program tries to access it which says says "Undefined relation" for the config table.
My questions are:
Why sqlworkbench where I had previously run the create statement recognizes the table while my java program cannot find it?
Where does the postgressql DBMS puts the tables I created? I don't see them neither in public nor in information schema.
NOTE:
I checked target postgres database and cannot see the table Config anywhere although SQL workbench can query it. Then I opened another SQL workbench instance and noticed that the table cannot be queried (i.e. not found). So, my conclusion is that PostgreSQL puts the table I created in the first running SQLBench instance into some location that is bound to that session. Another SQL Workbench instance or my java program is not bound to session, so cannot query the previously created table config.
The only "bloody location" that is session-local in PostgreSQL is the schema pg_temp, in other words: temporary tables. But your CREATE command does not display the keyword TEMP[ORARY]. Of course, as long as the transaction is not commited, nobody sees anything outside the transaction.
It's more likely you are seeing a switcheroo of hosts / databases / ports / or the schema search_path. A mixup with the mixed-case table name is a hot candidate, too. If you don't double-quote "Config", the table ends up all lower case in the system, so: config. If you later double quote the name, it won't match. The manual has the details.
Maybe the create failed on the extra trailing comma?
CREATE TABLE config (
id serial PRIMARY KEY,
pub_ip_range_low varchar(100),
pub_ip_range_high varchar(100) -- >> ,
);
I'm just getting started with PostgreSQL, and I'm new to database design.
I'm writing software in which I have various plugins that update a database. Each plugin periodically updates its own designated table in the database. So a plugin named 'KeyboardPlugin' will update the 'KeyboardTable', and 'MousePlugin' will update the 'MouseTable'. I'd like for my database to store these 'plugin-table' relationships while enforcing referential integrity. So ideally, I'd like a configuration table with the following columns:
Plugin-Name (type 'text')
Table-Name (type ?)
My software will read from this configuration table to help the plugins determine which table to update. Originally, my idea was to have the second column (Table-Name) be of type 'text'. But then, if someone mistypes the table name, or an existing relationship becomes invalid because of someone deleting a table, we have problems. I'd like for the 'Table-Name' column to act as a reference to another table, while enforcing referential integrity.
What is the best way to do this in PostgreSQL? Feel free to suggest an entirely new way to setup my database, different from what I'm currently exploring. Also, if it helps you answer my question, I'm using the pgAdmin tool to setup my database.
I appreciate your help.
I would go with your original plan to store the name as text. Possibly enhanced by additionally storing the schema name:
addin text
,sch text
,tbl text
Tables have an OID in the system catalog (pg_catalog.pg_class). You can get those with a nifty special cast:
SELECT 'myschema.mytable'::regclass
But the OID can change over a dump / restore. So just store the names as text and verify the table is there by casting it like demonstrated at application time.
Of course, if you use each tables for multiple addins it might pay to make a separate table
CREATE TABLE tbl (
,tbl_id serial PRIMARY KEY
,sch text
,name text
);
and reference it in ...
CREATE TABLE addin (
,addin_id serial PRIMARY KEY
,addin text
,tbl_id integer REFERENCES tbl(tbl_id) ON UPDATE CASCADE ON DELETE CASCADE
);
Or even make it an n:m relationship if addins have multiple tables. But be aware, as #OMG_Ponies commented, that a setup like this will require you to execute a lot of dynamic SQL because you don't know the identifiers beforehand.
I guess all plugins have a set of basic attributes and then each plugin will have a set of plugin-specific attributes. If this is the case you can use a single table together with the hstore datatype (a standard extension that just needs to be installed).
Something like this:
CREATE TABLE plugins
(
plugin_name text not null primary key,
common_int_attribute integer not null,
common_text_attribute text not null,
plugin_atttributes hstore
)
Then you can do something like this:
INSERT INTO plugins
(plugin_name, common_int_attribute, common_text_attribute, hstore)
VALUES
('plugin_1', 42, 'foobar', 'some_key => "the fish", other_key => 24'),
('plugin_2', 100, 'foobar', 'weird_key => 12345, more_info => "10.2.4"');
This creates two plugins named plugin_1 and plugin_2
Plugin_1 has the additional attributes "some_key" and "other_key", while plugin_2 stores the keys "weird_key" and "more_info".
You can index those hstore columns and query them very efficiently.
The following will select all plugins that have a key "weird_key" defined.
SELECT *
FROM plugins
WHERE plugin_attributes ? 'weird_key'
The following statement will select all plugins that have a key some_key with the value the fish:
SELECT *
FROM plugins
WHERE plugin_attributes #> ('some_key => "the fish"')
Much more convenient than using an EAV model in my opinion (and most probably a lot faster as well).
The only drawback is that you lose type-safety with this approach (but usually you'd lose that with the EAV concept as well).
You don't need an application catalog. Just add the application name to the keys of the table. This of course assumes that all the tables have the same structure. If not: use the application name for a table name, or as others have suggested: as a schema name( which also would allow for multiple tables per application).
EDIT:
But the real issue is of course that you should first model your data, and than build the applications to manipulate it. The data should not serve the code; the code should serve the data.