Script does not create tables but pgAdmin does - postgresql

I am running a script that creates a database, some tables with foreign keys and insert some data, but somehow creating the tables is not working, although it doesn't throw any error: I go to pgAdmin, look for the tables created and there's no one...
When I copy the text of my script and execute it into the Query Tool, it works fine and creates the tables.
Can you please explain me what I am doing wrong?
Script:
DROP DATABASE IF EXISTS test01 WITH (FORCE); --drops even if in use
CREATE DATABASE test01
WITH
OWNER = postgres
ENCODING = 'UTF8'
LC_COLLATE = 'German_Germany.1252'
LC_CTYPE = 'German_Germany.1252'
TABLESPACE = pg_default
CONNECTION LIMIT = -1
IS_TEMPLATE = False
;
CREATE TABLE customers
(
customer_id INT GENERATED ALWAYS AS IDENTITY,
customer_name VARCHAR(255) NOT NULL,
PRIMARY KEY(customer_id)
);
CREATE TABLE contacts
(
contact_id INT GENERATED ALWAYS AS IDENTITY,
customer_id INT,
contact_name VARCHAR(255) NOT NULL,
phone VARCHAR(15),
email VARCHAR(100),
PRIMARY KEY(contact_id),
CONSTRAINT fk_customer
FOREIGN KEY(customer_id)
REFERENCES customers(customer_id)
ON DELETE CASCADE
);
INSERT INTO customers(customer_name)
VALUES('BlueBird Inc'),
('Dolphin LLC');
INSERT INTO contacts(customer_id, contact_name, phone, email)
VALUES(1,'John Doe','(408)-111-1234','john.doe#bluebird.dev'),
(1,'Jane Doe','(408)-111-1235','jane.doe#bluebird.dev'),
(2,'David Wright','(408)-222-1234','david.wright#dolphin.dev');
I am calling the script from a Windows console like this:
"C:\Program Files\PostgreSQL\15\bin\psql.exe" -U postgres -f "C:\Users\my user name\Desktop\db_create.sql" postgres
My script is edited in Notepad++ and saved with Encoding set to UTF-8 without BOM, as per a suggestion found here

I see you are using -U postgres command line parameter, and also using database name as last parameter (postgres).
So all your SQL commands was executed while you are connected to postgres database. Of course, CREATE DATABASE command did creation of test01 database, but CREATE TABLE and INSERT INTO did executed not for test01 database, but for postgres database, and all your tables are in postgres database, but not in test01.
You need to split your SQL script into 2 scripts (files): first for 'CREATE DATABASE', second for the rest of.
You need to execute first script as before, like
psql.exe -U postgres -f "db_create_1.sql" postgres
And for second one need to choose the database which was created at 1st step, like
psql.exe -U postgres -f "db_create_2.sql" test01

Related

Postgresql migrate schema with large objects included, but only for this schema

I have a super large database in postgresql 13, the size is 1 TB and I need to migrate only one schema to another database, the problem is that this schema has blobs. So if I migrate with pg_dump and the --blobs property, the command makes a backup of all the blobs in the database and I only want it to store only the blobs of this scheme. is this possible? this is the command i am executing to do the dump.
pg_dump --host=$HOST_ORIGIN --dbname=$BD_ORIGIN --port=$BD_PORT_ORIGIN --username=$BD_USER_ORIGIN --schema=$SCHEMA --no-privileges --blobs -v -Fc > schema.sql
I had this problem and I solved it like this:
I created a second database with the total data.
And I copied only the oids from one DB to another using:
create extension dblink;
INSERT INTO pg_catalog.pg_largeobject(loid, pageno, data)
select loid, pageno, data from dblink('dbname=dbOrigin', 'select * from pg_largeobject where loid in
(select oid from schema.table)') as t
( loid oid,
pageno integer,
data bytea
);
INSERT INTO pg_catalog.pg_largeobject_metadata(oid, lomowner, lomacl)
select oid, lomowner, lomacl from dblink('dbname=DBOrigin', 'select oid, lomowner, lomacl from pg_largeobject_metadata where oid in
(select oid from schema.table)') as t
( oid oid,
lomowner oid,
lomacl aclitem[]
);

PostgreSql foreign table select fails due to special characters rows

I just set up a new foreign table and it works as intended if I just select the "ID" (integer) field.
When I add the "Description"(text) field and try to select the table, it fails with this error message:
utf-8 'Codec cannot decode byte 0xfc in position 10: invalid start byte
After checking the remote table, I found that "Description" contains special characters like: "ö, ü, ä"
What can i do to fix this?
Table definitions (Only first 2 rows)
Remote table:
CREATE TABLE test (
[Id] [char](8) NOT NULL,
[Description] [nvarchar](50) NOT NULL
)
Foreign table:
Create Foreign Table "Test" (
"Id" Char(8),
"Description" VarChar(50)
) Server "Remote" Options (
schema_name 'dbo', table_name 'test'
);
Additional information:
Foreign data wrapper: tds_fdw
Local server: Postgres 12, encoding: UTF8
Remote server: Sql Server, encoding: Latin1_General_CI_AS
As Laurenz Albe suggested in the comments, I created a freetds.conf in my PostgreSQL folder with the following content:
[global]
tds version = auto
client charset = UTF-8
Don't forget to set the path to the configuration file in the environment variable FREETDS.
Powershell:
[System.Environment]::SetEnvironmentVariable('FREETDS','C:\Program Files\PostgreSQL\12',[System.EnvironmentVariableTarget]::Machine)

Optimize bulk insert speed for PostgreSQL

I have a PostgreSQL database on an Amazon EC2 instance, and I am trying to seed the database with 100M lines of data (I have 10 files of 10M lines each). I used the secure copy (scp) command to move the csv files into the EC2 instance. When I try to copy the csv files into the database, it is taking way too much time. Is there a way to fasten the procedure?
Here is my schema.sql file:
DROP DATABASE IF EXISTS reviews_db;
CREATE DATABASE reviews_db;
\c reviews_db;
CREATE TABLE reviews (
id INT PRIMARY KEY NOT NULL,
houseId INT NOT NULL,
name VARCHAR(30) NOT NULL,
picture VARCHAR(55) NOT NULL,
reviewText TEXT NOT NULL,
reviewDate VARCHAR (15) NOT NULL,
accuracyRating INT NOT NULL,
locationRating INT NOT NULL,
communicationRating INT NOT NULL,
checkinRating INT NOT NULL,
cleanlinessRating INT NOT NULL,
valueRating INT NOT NULL,
overallRating DECIMAL NOT NULL
);
CREATE INDEX ON reviews (houseId);
Then, on my EC2 instance, I run this command to seed the database :
pv ./reviews1.csv | psql -U postgres -d reviews_db -c "COPY reviews FROM STDIN with (FORMAT csv);"
Note: my reviews1.csv file is 3.1GB, and it is uploading at a speed of 200kb/s

Automate Native Range Partitioning in PostgreSQL 10 for Zabbix 3.4

I would like to automate the process of partitioning a Zabbix 3.4 Database using PostgreSQL's Native Range Partitioning.
Would it be wiser to write a SQL function to perform the following below or to use a shell/python script?
Make sure at least one partition is created ahead of what's needed.
Delete any partition older than x weeks/months; for history 7 days and for trends 1 year
The following below is the solution I came up with for transitioning to PSQL 10 Native Range Partitioning from a PSQL 9.4 populated database with no partitioning.
A. Create a Zabbix Empty PSQL 10 Database.
Ensure you first create an empty Zabbix PSQL 10 DB.
# su postgres
postgres#<>:~$ createuser -P -s -e zabbix
postgres#<>:~$ psql
postgres# create database zabbix;
postgres# grant all privileges on database zabbix to zabbix;
B. Create tables & Native Range Partitions on clock column
Create the tables in the Zabbix DB and implement native range partitioning for the clock column. Below is an example of a manual SQL script that can be fun for history table. Perform this for all the history tables you want to partition via range.
zabbix=# CREATE TABLE public.history
(
itemid bigint NOT NULL,
clock integer NOT NULL DEFAULT 0,
value numeric(20,0) NOT NULL DEFAULT (0)::numeric,
ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);
zabbix=# CREATE TABLE public.history_old PARTITION OF public.history
FOR VALUES FROM (MINVALUE) TO (1522540800);
zabbix=# CREATE TABLE public.history_y2018m04 PARTITION OF public.history
FOR VALUES FROM (1522540800) TO (1525132800);
zabbix=# CREATE TABLE public.history_y2018m05 PARTITION OF public.history
FOR VALUES FROM (1525132800) TO (1527811200);
zabbix=# CREATE INDEX ON public.history_old USING btree (itemid, clock);
zabbix=# CREATE INDEX ON public.history_y2018m04 USING btree (itemid, clock);
zabbix=# CREATE INDEX ON public.history_y2018m05 USING btree (itemid, clock);
C. Automate it!
I used a shell script because it is one of the simplest ways to deal with creating new partitions in PSQL 10. Make sure you're always at least one partition ahead of what's needed.
Let's call the script auto_history_tables_monthly.sh.
On a Debian 8 Flavor OS that runs PSQL 10 ensure the script is in a certain directory (I used /usr/local/bin) with correct permissions (chown postgres:postgres /usr/local/bin/auto_history_tables_monthly.sh) and make it executable (chmod u+x /usr/local/bin/auto_history_tables_monthly.sh as postgres user).
Create a cron job (crontab -e) for postgres user with following:
0 0 1 * * /usr/local/bin/auto_history_tables_monthly.sh | psql -d zabbix
This will run the shell script the first of every month.
Below is the script. It uses the date command to leverage the UTC epoch value. It creates a table a month in advance and drops a partition 2 months older. This appears to work well in conjunction with a 31 days of retention for history that is customized to my needs. Ensure the PSQL 10 DB is on UTC time for this use case.
#!/bin/bash
month_diff () {
year=$1
month=$2
delta_month=$3
x=$((12*$year+10#$month-1))
x=$((x+$delta_month))
ry=$((x/12))
rm=$(((x % 12)+1))
printf "%02d %02d\n" $ry $rm
}
month_start () {
year=$1
month=$2
date '+%s' -d "$year-$month-01 00:00:00" -u
}
month_end () {
year=$1
month=$2
month_start $(month_diff $year $month 1)
}
# Year using date
current_year=$(date +%Y)
current_month=$(date +%m)
# Math
next_date=$(month_diff $current_year $current_month 1)
next_year=$(echo $next_date|sed 's/ .*//')
next_month=$(echo $next_date|sed 's/.* //')
start=$(month_start $next_date)
end=$(month_end $next_date)
#next_month_table="public.history_y${next_year}m${next_month}"
# Create next month table for history, history_uint, history_str, history_log, history_text
sql="
CREATE TABLE IF NOT EXISTS public.history_y${next_year}m${next_month} PARTITION OF public.history
FOR VALUES FROM ($start) TO ($end);
\nCREATE TABLE IF NOT EXISTS public.history_uint_y${next_year}m${next_month} PARTITION OF public.history_uint
FOR VALUES FROM ($start) TO ($end);
\nCREATE TABLE IF NOT EXISTS public.history_str_y${next_year}m${next_month} PARTITION OF public.history_str
FOR VALUES FROM ($start) TO ($end);
\nCREATE TABLE IF NOT EXISTS public.history_log_y${next_year}m${next_month} PARTITION OF public.history_log
FOR VALUES FROM ($start) TO ($end);
\nCREATE TABLE IF NOT EXISTS public.history_text_y${next_year}m${next_month} PARTITION OF public.history_text
FOR VALUES FROM ($start) TO ($end);
\nCREATE INDEX on public.history_y${next_year}m${next_month} USING btree (itemid, clock);
\nCREATE INDEX on public.history_uint_y${next_year}m${next_month} USING btree (itemid, clock);
\nCREATE INDEX on public.history_str_y${next_year}m${next_month} USING btree (itemid, clock);
\nCREATE INDEX on public.history_log_y${next_year}m${next_month} USING btree (itemid, clock);
\nCREATE INDEX on public.history_text_y${next_year}m${next_month} USING btree (itemid, clock);
"
echo -e $sql
# Math
prev_date=$(month_diff $current_year $current_month -2)
prev_year=$(echo $prev_date|sed 's/ .*//')
prev_month=$(echo $prev_date|sed 's/.* //')
# Drop last month table for history, history_uint, history_str, history_log, history_text
sql="
DROP TABLE public.history_y${prev_year}m${prev_month};
\nDROP TABLE public.history_uint_y${prev_year}m${prev_month};
\nDROP TABLE public.history_str_y${prev_year}m${prev_month};
\nDROP TABLE public.history_log_y${prev_year}m${prev_month};
\nDROP TABLE public.history_text_y${prev_year}m${prev_month};
"
echo -e $sql
D. Then dump the data from the old database within. I used pg_dump/pg_restore.
I'm sure there are more complex solutions out there but I found this to be simplest for the needs of autopartitioning the Zabbix Database using the PostgreSQL 10 Native Range Partitioning functionality.
Please let me know if you need more details.
I have written detailed notes on using PostgreSQL version 11 together with pgpartman as the mechanism for native table partitioning with Zabbix (version 3.4 as of this writing).
zabbix-postgres-partitioning

CSVSQL - trying to force db-schema without success after getting (in table 'blabla', column 'xyz'): VARCHAR requires a length on dialect mysql

I'm trying to build a table with csvsql.
When I use command:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name file.csv
I get the error:
(in table 'blabla', column 'xyz'): VARCHAR requires a length on dialect mysql
I've then tried to build a database schema and force it with --db-schema flag,
The db-schema format is:
CREATE TABLE table_name (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`x` varchar(29) DEFAULT NULL,
`y` int(10) NOT NULL DEFAULT '0',
`z` BOOL NOT NULL,
PRIMARY KEY (`id`),
KEY `indexed` (`indexed`)
);
but I still get the same error.
The complete command with db-schema is:
csvsql --db mysql://user:password#localhost:3306/database_name --table table_name --db-schema db_schema_filename csvfile.csv
I've read the manual for csvkit, but I don't get what I'm doing wrong.
This command should print the conversion result right?
Can someone please help?
Thank you.
Well, found the solution in the github.
https://github.com/wireservice/csvkit/issues/758#issue-201924611
After update from github, no more errors and tables are created normaly.