psql SQL Interpolation in a code block - postgresql

In some of my scripts I use SQL Interpolation feature of psql utility:
basic.sql:
update :schema.mytable set ok = true;
> psql -h 10.0.0.1 -U postgres -f basic.sql -v schema=myschema
Now I need bit more complicated scenario. I need to specify schema name (and desirebly some other things) inside PL/pgSQL code block:
pg.sql
do
$$
begin
update :schema.mytable set ok = true;
end;
$$
But unfortunately this does not work, since psql does not replace :variables inside $$.
Is there a way to workaround it in general? Or more specifically, how to substitute schema names into pgSQL code block or function definition?

in your referenced docs:
Variable interpolation will not be performed within quoted SQL
literals and identifiers. Therefore, a construction such as ':foo'
doesn't work to produce a quoted literal from a variable's value (and
it would be unsafe if it did work, since it wouldn't correctly handle
quotes embedded in the value).
it does not matter if quotes are double dollar sign or single quote - it wont work, eg:
do
'
begin
update :schema.mytable set ok = true;
end;
'
ERROR: syntax error at or near ":"
to pass variable into quoted statement other way you can try using shell variables, eg:
MacBook-Air:~ vao$ cat do.sh; export schema_name='smth' && bash do.sh
psql -X so <<EOF
\dn+
do
\$\$
begin
execute format ('create schema %I','$schema_name');
end;
\$\$
;
\dn+
EOF
List of schemas
Name | Owner | Access privileges | Description
----------+----------+-------------------+------------------------
public | vao | vao=UC/vao +| standard public schema
| | =UC/vao |
schema_a | user_old | |
(2 rows)
DO
List of schemas
Name | Owner | Access privileges | Description
----------+----------+-------------------+------------------------
public | vao | vao=UC/vao +| standard public schema
| | =UC/vao |
schema_a | user_old | |
smth | vao | |
(3 rows)

Related

How to access passed variables in psql command with -v option

I have a bash program that is executing a postgreSQL statement:
psql -q -d $DB --echo-all --set ON_ERROR_STOP=on -f $APP_BASE_PATH/$PROJ/$SCRIPT_NAME -v job_end_bndry_ts="${job_end_bndry_ts}" -v job_exec_uid=${JOB_UID} > ${LOGPATH}/${MOD_NM}_${LOGTIME}.log 2> >(tee -a $errmsg >&1)
note the -v job_exec_uid=${JOB_UID}
This command will run a file that stages the data from a temporary external table:
CREATE READABLE EXTERNAL TEMPORARY TABLE json_ext (json_data text)
LOCATION ('<location>') FORMAT 'TEXT' (DELIMITER 'OFF' null E'' escape E'\t')
LOG ERRORS SEGMENT REJECT LIMIT 100 PERCENT;
Here is the file that the bash script runs to populate the staging table:
INSERT INTO schema.staging_table
(
JSON_DATAREC_FL,
JOBEXEC_UID,
JSON_FL_ID,
INSRT_TS
)
SELECT
json_data AS JSON_DATA,
'job_exec_uid'::integer AS JOBEXEC_UID,
(JSON_DATA::json#>>'{refNumber}') AS JSON_FILE_ID,
now() INSRT_TS
FROM json_ext
;
When I run the bash script, I get the following error:
ERROR: invalid input syntax for integer: "job_exec_uid"
Here is the DDL for the staging table:
Column | Type | Modifiers
-----------------+-----------------------------+-----------
json_fl_id | character varying(100) | not null
jobexec_uid | integer |
json_datarec_fl | text |
insrt_ts | timestamp without time zone | not null
The psql variable is evaluated by syntax :variable or :'varable for literal values.
[pavel#localhost postgresql.master]$ psql -v xxx=10 -v yyy='Ahoj'
Assertions: on
psql (16devel)
Type "help" for help.
(2022-10-10 19:42:56) postgres=# select :xxx, :'yyy';
┌──────────┬──────────┐
│ ?column? │ ?column? │
╞══════════╪══════════╡
│ 10 │ Ahoj │
└──────────┴──────────┘
(1 row)
Here is documentation. See Advanced features/Variables section.

Why I am not able to get the specific column in psql

I have a table in my psql db called cls and user called cls
then tries to get the specific column from the existing table [name: test] but I am not able to retrieve the table.
Snippets as below:
psql -U cls
cls # select * from test;
name | 
 ip | 
 user | 
 password | 
 group | 
 created_on
-----------+--------+------+--------------+--------+----------------------------
server | 1.1.1.1 | test | pwd | gp1 | 2022-08-04 13:55:00.765548
cls # select ip from test where name='server';
LINE 1: select ip from test where name='server';
^
HINT: Perhaps you meant to reference the column "test.
ip".
cls # select test.ip from test where name='server';
LINE 1: select ip from test where name='server';
^
HINT: Perhaps you meant to reference the column "test.
ip".
cls # select t.ip from test t;
LINE 1: select t.ip from test t;
^
HINT: Perhaps you meant to reference the column "t.
ip".
I tried double quotes and single quotes but no luck.
As the error message says, your column isn't called ip, it's called 
ip - notice the "funny" character before the i.

What is the simplest way to migrate data from MySQL to DB2

I need to migrate data from MySQL to DB2. Both DBs are up and running.
I tried to mysqldump with --no-create-info --extended-insert=FALSE --complete-insert and with a few changes on the output (e.g. change ` to "), I get to a satisfactory result but sometimes I have weird exceptions, like
does not have an
ending string delimiter. SQLSTATE=42603
Ideally I would want to have a routine that is as general as possible, but as an example here, let's say I have a DB2 table that looks like:
db2 => describe table "mytable"
Data type Column
Column name schema Data type name Length Scale Nulls
------------------------------- --------- ------------------- ---------- ----- ------
id SYSIBM BIGINT 8 0 No
name SYSIBM VARCHAR 512 0 No
2 record(s) selected.
Its MySQL counterpart being
mysql> describe mytable;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| name | varchar(512) | NO | | NULL | |
+-------+--------------+------+-----+---------+----------------+
2 rows in set (0.01 sec)
Let's assume the DB2 and MySQL databases are called mydb.
Now, if I do
mysqldump -uroot mydb mytable --no-create-info --extended-insert=FALSE --complete-insert | # mysldump, with options (see below): # do not output table create statement # one insert statement per record# ouput table column names
sed -n -e '/^INSERT/p' | # only keep lines beginning with "INSERT"
sed 's/`/"/g' | # replace ` with "
sed 's/;$//g' | # remove `;` at end of insert query
sed "s/\\\'/''/g" # replace `\'` with `''` , see http://stackoverflow.com/questions/2442205/how-does-one-escape-an-apostrophe-in-db2-sql and http://stackoverflow.com/questions/2369314/why-does-sed-require-3-backslashes-for-a-regular-backslash
, I get:
INSERT INTO "mytable" ("id", "name") VALUES (1,'record 1')
INSERT INTO "mytable" ("id", "name") VALUES (2,'record 2')
INSERT INTO "mytable" ("id", "name") VALUES (3,'record 3')
INSERT INTO "mytable" ("id", "name") VALUES (4,'record 4')
INSERT INTO "mytable" ("id", "name") VALUES (5,'" "" '' '''' \"\" ')
This ouput can be used as a DB2 query and it works well.
Any idea on how to solve this more efficiently/generally? Any other suggestions?
After having played around a bit I came with the following routine which I believe to be fairly general, robust and scalable.
1 Run the following command:
mysqldump -uroot mydb mytable --no-create-info --extended-insert=FALSE --complete-insert | # mysldump, with options (see below): # do not output table create statement # one insert statement per record# ouput table column names
sed -n -e '/^INSERT/p' | # only keep lines beginning with "INSERT"
sed 's/`/"/g' | # replace ` with "
sed -e 's/\\"/"/g' | # replace `\"` with `#` (mysql escapes double quotes)
sed "s/\\\'/''/g" > out.sql # replace `\'` with `''` , see http://stackoverflow.com/questions/2442205/how-does-one-escape-an-apostrophe-in-db2-sql and http://stackoverflow.com/questions/2369314/why-does-sed-require-3-backslashes-for-a-regular-backslash
Note: here unlike in the question ; are not being removed.
2 upload the file to DB2 server
scp out.sql user#myserver:out.sql
3 run queries from the file
db2 -tvsf /path/to/query/file/out.sql

Getting an error of 'Token Unknown' while Execute Stored Procedures

I'm new in learning stored procedures in SQL.
I want to create a stored procedure for inserting values from automatic data by calculation.
Table Attendance:
EMPL_KODE |EMPL_NAME |DATE_IN |TIME_IN |TIME_OUT|TIME_IN |TIME_OUT
001 | Michel |25.04.2016 |06:50 |15:40 | |
002 | Clara |25.04.2016 |06:15 |15:43 | |
003 | Rafael |25.04.2016 |06:25 |15:45 | |
001 | Michel |26.04.2016 |06:23 |15:42 | |
002 | Clara |26.04.2016 |06:10 |15:41 | |
003 | Rafael |26.04.2016 |06:30 |15:42 | |
001 | Michel |27.04.2016 |06:33 |15:42 | |
002 | Clara |27.04.2016 |06:54 |15:44 | |
003 | Rafael |27.04.2016 |07:00 |15:45 | |
I want to fill TIME_IN and TIME_OUT values automatically by creating a stored procedure. Here is the code :
CREATE PROCEDURE InsertTotalEmployee
#TOTAL_MINUTES int,
#TOTAL_HOURS float
AS
BEGIN
INSERT INTO ATTENDANCE (TOTAL_MINUTES, TOTAL_HOURS)
VALUES (
SELECT
DATEDIFF(MINUTE, ATTENDANCE.TIME_IN, ATTENDANCE.TIME_OUT),
DATEDIFF(MINUTE, ATTENDANCE.TIME_IN, ATTENDANCE.TIME_OUT) / 60.0
)
END
After I write and execute my statement, a message error occurs:
Token unknown - line 2, column 5 #
I run the code using Flamerobin.
It looks like you are trying to use Microsoft SQL Server syntax in Firebird, that is not going to work.
For one, the # is not allowed like that in identifiers (unless you use double quotes around them), and the list of parameters must be enclosed in parentheses.
See the syntax of CREATE PROCEDURE. You need to change it to:
CREATE PROCEDURE InsertTotalEmployee(TOTAL_MINUTES int, TOTAL_HOURS float)
You also might want to change the datatype float to double precision, and the body of your stored procedure seems to be incomplete because you are selecting from nothing (a select requires a table to select from), and are missing a semicolon at the end of the statement.
All in all I suggest you study the Firebird language reference, then try to create a functioning insert and only then create a stored procedure around it.
Also note that when creating a stored procedure in Flamerobin, that you must switch statement terminators using set term otherwise Flamerobin can't send the stored procedure correctly, see also the first section in Procedural SQL (PSQL) Statements.

How to Check for Two Columns and Query Every Table When They Exist?

I'm interested in doing a COUNT(*), SUM(LENGTH(blob)/1024./1024.), and ORDER BY SUM(LENGTH(blob)) for my entire database when column 'blob' exists. For tables where synchlevels does not exist, I still want the output. I'd like to GROUP BY that column:
Example
+--------+------------+--------+-----------+
| table | synchlevel | count | size_mb |
+--------+------------+--------+-----------+
| tableA | 0 | 924505 | 3013.47 |
| tableA | 7 | 981 | 295.33 |
| tableB | 6 | 1449 | 130.50 |
| tableC | 1 | 64368 | 68.43 |
| tableD | NULL | 359 | .54 |
| tableD | NULL | 778 | .05 |
+--------+------------+--------+-----------+
I would like to do a pure SQL solution, but I'm having a bit of difficulty with that. Currently, I'm wrapping some SQL into BASH.
#!/bin/bash
USER=$1
DBNAME=$2
function psql_cmd(){
cmd=$1
prefix='\pset border 2 \\ '
echo $prefix $cmd | psql -U $USER $DBNAME | grep -v "Border\| row"
}
function synchlevels(){
echo "===================================================="
echo " SYNCH LEVEL STATS "
echo "===================================================="
tables=($(psql -U $USER -tc "SELECT table_name FROM information_schema.columns
WHERE column_name = 'blob';" $DBNAME))
for table in ${tables[#]}; do
count_size="SELECT t.synchlevel,
COUNT(t.blob) AS count,
to_char(SUM(LENGTH(t.blob)/1024./1024.),'99999D99') AS size_mb
FROM $table AS t
GROUP BY t.synchlevel
ORDER BY SUM(LENGTH(t.blob)) DESC;"
echo $table
psql_cmd "$count_size"
done
echo "===================================================="
}
I could extend this by creating a second tables BASH array of tables which have the 'synchlevel' column, compare and use that list to run through the SQL, but I was wondering if there was a way I could just do the SQL portion purely in SQL without resorting to making these lists in BASH and doing the comparisons externally. i.e. I want to avoid needing to externally loop through the tables and making numerous queries in tables=($(psql -U $USER....
I've tried the following SQL to test on a table where I know the column doesn't exist...
SELECT
CASE WHEN EXISTS(SELECT * FROM information_schema.columns
WHERE column_name = 'synchlevel'
AND table_name = 'archivemetadata')
THEN synchlevel
END,
COUNT(blob) AS count,
to_char(SUM(LENGTH(blob)/1024./1024.),'99999D99') AS size_mb
FROM archivemetadata, information_schema.columns AS info
WHERE info.column_name = 'blob'
However, it fails on THEN synchlevel for tables where it doesn't exist. It seems really simple to do, but I just can't seem to find a way to do this which doesn't require either:
Resorting to external array comparisons in BASH.
Can be done, but I'd like to simplify my solution rather than add another layer.
Creating PL/PGSQL functions.
This script is really just to help with some database data analysis for improving performance in a third-party software. We are not a shop of DB Admins, so I would prefer not to dive into PL/PGSQL as that would require more folks from our shop to also become acquainted with the language in order to support the script. Again, simplicity is the motivation here.
Postgresql 8.4 is the engine. (We cannot upgrade due to security constraints by an overseeing IT body.)
Thanks for any suggestions you might have!
The following is untested, but how about creating some dynamic sql in one psql session and piping it to another?
psql -d <yourdb> -qtAc "
select 'select ' || (case when info.column_name = 'synchlevel' then 'synchlevel,' else '' end) ||
'count(*) as cnt,' ||
'to_char(SUM(LENGTH(blob)::NUMERIC/1024/1024),''99999D99'') AS size_mb' ||
'from ' || info.table_name ||
(case when info.column_name = 'synchlevel' then ' group by synchlevel order by synchlevel' else '' end)
from information_schema.columns as info
where info.table_name IN (select distinct table_name from information_schema.columns where column_name = 'blob')" | psql -d <yourdb>