I have a bash program that is executing a postgreSQL statement:
psql -q -d $DB --echo-all --set ON_ERROR_STOP=on -f $APP_BASE_PATH/$PROJ/$SCRIPT_NAME -v job_end_bndry_ts="${job_end_bndry_ts}" -v job_exec_uid=${JOB_UID} > ${LOGPATH}/${MOD_NM}_${LOGTIME}.log 2> >(tee -a $errmsg >&1)
note the -v job_exec_uid=${JOB_UID}
This command will run a file that stages the data from a temporary external table:
CREATE READABLE EXTERNAL TEMPORARY TABLE json_ext (json_data text)
LOCATION ('<location>') FORMAT 'TEXT' (DELIMITER 'OFF' null E'' escape E'\t')
LOG ERRORS SEGMENT REJECT LIMIT 100 PERCENT;
Here is the file that the bash script runs to populate the staging table:
INSERT INTO schema.staging_table
(
JSON_DATAREC_FL,
JOBEXEC_UID,
JSON_FL_ID,
INSRT_TS
)
SELECT
json_data AS JSON_DATA,
'job_exec_uid'::integer AS JOBEXEC_UID,
(JSON_DATA::json#>>'{refNumber}') AS JSON_FILE_ID,
now() INSRT_TS
FROM json_ext
;
When I run the bash script, I get the following error:
ERROR: invalid input syntax for integer: "job_exec_uid"
Here is the DDL for the staging table:
Column | Type | Modifiers
-----------------+-----------------------------+-----------
json_fl_id | character varying(100) | not null
jobexec_uid | integer |
json_datarec_fl | text |
insrt_ts | timestamp without time zone | not null
The psql variable is evaluated by syntax :variable or :'varable for literal values.
[pavel#localhost postgresql.master]$ psql -v xxx=10 -v yyy='Ahoj'
Assertions: on
psql (16devel)
Type "help" for help.
(2022-10-10 19:42:56) postgres=# select :xxx, :'yyy';
┌──────────┬──────────┐
│ ?column? │ ?column? │
╞══════════╪══════════╡
│ 10 │ Ahoj │
└──────────┴──────────┘
(1 row)
Here is documentation. See Advanced features/Variables section.
Related
I have a table in my psql db called cls and user called cls
then tries to get the specific column from the existing table [name: test] but I am not able to retrieve the table.
Snippets as below:
psql -U cls
cls # select * from test;
name |
ip |
user |
password |
group |
created_on
-----------+--------+------+--------------+--------+----------------------------
server | 1.1.1.1 | test | pwd | gp1 | 2022-08-04 13:55:00.765548
cls # select ip from test where name='server';
LINE 1: select ip from test where name='server';
^
HINT: Perhaps you meant to reference the column "test.
ip".
cls # select test.ip from test where name='server';
LINE 1: select ip from test where name='server';
^
HINT: Perhaps you meant to reference the column "test.
ip".
cls # select t.ip from test t;
LINE 1: select t.ip from test t;
^
HINT: Perhaps you meant to reference the column "t.
ip".
I tried double quotes and single quotes but no luck.
As the error message says, your column isn't called ip, it's called
ip - notice the "funny" character before the i.
I have a PostgreSQL query as below which is running fine . I am calling it from a shell script as below
Result=$(psql -U username -d database -t -c
$'SELECT round(sum(i.total), 2) AS "ROUND(sum(i.total), 2)"
FROM invoice i
WHERE i.create_datetime = '2019-03-01 00:00:00-06'
AND i.is_review = '1' AND i.user_id != 60;')
now I want the value which I have hard coded as i.create_datetime = '2019-03-01 00:00:00-06' to replace it with a variable date value.
I have tried two ways
way 1:
Result=$(psql -U username -d database -t -c
$'WITH var(reviewMonth) as (values(\'$reviewMonth\'))
SELECT round(sum(i.total),2) AS "ROUND(sum(i.total),2)"
FROM var,invoice i
WHERE i.create_datetime = var.reviewMonth::timestamp
AND i.is_review = \'1\' AND i.user_id != 60;')
and
way 2:
Result=$(psql -U username -d database -t -c
$'SELECT round(sum(i.total),2) AS "ROUND(sum(i.total),2)"
FROM invoice i
WHERE i.create_datetime = \'$reviewMonth\'
AND i.is_review = \'1\' AND i.user_id != 60;')
But both way it's throwing error
way 1 throwing error as :
ERROR: operator does not exist: timestamp with time zone = text
way 2 throwing error as :
ERROR: invalid input syntax for type timestamp with time zone: "$reviewMonth"
Please suggest what should be my approach.
You should try using the psql variables. Here's an example:
# Put the query in a file, with the variable TSTAMP:
> echo "SELECT :'TSTAMP'::timestamp with time zone;" > query.sql
> export TSTAMP='2019-03-01 00:00:00-06'
> RESULT=$(psql -U postgres -t --variable=TSTAMP="$TSTAMP" -f query.sql )
> echo $RESULT
2019-03-01 06:00:00+00
Note how we format the string literal substitution in the query: :'TSTAMP'
You could also do the substitution yourself. Here's an example using a heredoc:
> export TSTAMP='2019-03-01 00:00:01-06'
> RESULT=$(psql -U postgres -t << EOF
SELECT '$TSTAMP'::timestamp with time zone;
EOF
)
> echo $RESULT
2019-03-01 06:00:01+00
In this case, we aren't using psql's variable substitution, so we have to quote the variable like '$TSTAMP' . Using a heredoc makes the quoting much simpler than using -c because you aren't trying to quote the whole command.
EDIT: more examples because it appears this wasn't clear enough. TSTAMP does not have to be hard coded, it's just a bash variable than can be set like any other bash variable.
> TSTAMP=$(date -d 'now' +'%Y-%m-01 00:00:00')
> RESULT=$(psql -U postgres -t << EOF
SELECT '$TSTAMP'::timestamp with time zone;
EOF
)
> echo $RESULT
2019-06-01 00:00:00+00
However, if you're really just looking for the start of the month, there's no need for shell variables at all
> RESULT=$(psql -U postgres -t << EOF
SELECT date_trunc('month', now());
EOF
)
> echo $RESULT
2019-06-01 00:00:00+00
Assume the following table and custom range type:
create table booking (
identifier integer not null primary key,
room uuid not null,
start_time time without time zone not null,
end_time time without time zone not null
);
create type timerange as range (subtype = time);
In PostgreSQL v10, you can do:
alter table booking add constraint overlapping_times
exclude using gist
(
room with =,
timerange(start_time, end_time) with &&
);
In PostgreSQL v9.5/v9.6, you have to manually cast the uuid column as gist_btree does not support uuid:
alter table booking add constraint overlapping_times
exclude using gist
(
(room::text) with =,
timerange(start_time, end_time) with &&
);
I would like to support v9.5, v9.6 and v10 for my customers. Is there a way to conditionally add the above constraint in the same .sql file, depending on the version of the current database?
You can use dynamic sql
For example:
do
$block$
declare
l_version text;
begin
select setting into l_version from pg_settings where name = 'server_version';
execute
format(
$script$
alter table booking add constraint overlapping_times
exclude using gist
(
%s with =,
timerange(start_time, end_time) with &&
)
$script$,
case when (l_version like '9.5.%' or l_version like '9.6.%') then '(room::text)' else 'room' end
)
;
end;
$block$
language plpgsql;
Here is a proof of concept:
#!/bin/sh
#THE_HOST="192.168.0.104"
THE_HOST="192.168.0.101"
get_version ()
{
psql -t -h ${THE_HOST} -U postgres postgres <<OMG | awk -e '{ print $2; }'
select version();
OMG
}
# ############################################################################
# main
#
# - Connect to database to retrieve version
# - use the retrieved version to create a symlink "versioned" to
# one of our subdirs
# - call an sql script that includes this symlink/some.sql
# symlinking to a non-existing directory will cause a dead link
# (, and the script to fail.)
# ergo: there should be a subdir "verX.Y.Z" for every supported version X.Y.Z
# ############################################################################
pg_version=`get_version`
#echo "version=${pg_version}"
rm versioned
ln -fs "ver${pg_version}" versioned
if [ -f versioned/alter.sql ]; then
echo created link versioned to "ver${pg_version}"
else
echo "version ${pg_version} not supported today..."
echo "Failed!"
exit 1
fi
psql -t -h ${THE_HOST} -U postgres postgres <<LETS_GO
\i common/create.sql
\i versioned/alter.sql
\echo done!
LETS_GO
#eof
In some of my scripts I use SQL Interpolation feature of psql utility:
basic.sql:
update :schema.mytable set ok = true;
> psql -h 10.0.0.1 -U postgres -f basic.sql -v schema=myschema
Now I need bit more complicated scenario. I need to specify schema name (and desirebly some other things) inside PL/pgSQL code block:
pg.sql
do
$$
begin
update :schema.mytable set ok = true;
end;
$$
But unfortunately this does not work, since psql does not replace :variables inside $$.
Is there a way to workaround it in general? Or more specifically, how to substitute schema names into pgSQL code block or function definition?
in your referenced docs:
Variable interpolation will not be performed within quoted SQL
literals and identifiers. Therefore, a construction such as ':foo'
doesn't work to produce a quoted literal from a variable's value (and
it would be unsafe if it did work, since it wouldn't correctly handle
quotes embedded in the value).
it does not matter if quotes are double dollar sign or single quote - it wont work, eg:
do
'
begin
update :schema.mytable set ok = true;
end;
'
ERROR: syntax error at or near ":"
to pass variable into quoted statement other way you can try using shell variables, eg:
MacBook-Air:~ vao$ cat do.sh; export schema_name='smth' && bash do.sh
psql -X so <<EOF
\dn+
do
\$\$
begin
execute format ('create schema %I','$schema_name');
end;
\$\$
;
\dn+
EOF
List of schemas
Name | Owner | Access privileges | Description
----------+----------+-------------------+------------------------
public | vao | vao=UC/vao +| standard public schema
| | =UC/vao |
schema_a | user_old | |
(2 rows)
DO
List of schemas
Name | Owner | Access privileges | Description
----------+----------+-------------------+------------------------
public | vao | vao=UC/vao +| standard public schema
| | =UC/vao |
schema_a | user_old | |
smth | vao | |
(3 rows)
I need to migrate data from MySQL to DB2. Both DBs are up and running.
I tried to mysqldump with --no-create-info --extended-insert=FALSE --complete-insert and with a few changes on the output (e.g. change ` to "), I get to a satisfactory result but sometimes I have weird exceptions, like
does not have an
ending string delimiter. SQLSTATE=42603
Ideally I would want to have a routine that is as general as possible, but as an example here, let's say I have a DB2 table that looks like:
db2 => describe table "mytable"
Data type Column
Column name schema Data type name Length Scale Nulls
------------------------------- --------- ------------------- ---------- ----- ------
id SYSIBM BIGINT 8 0 No
name SYSIBM VARCHAR 512 0 No
2 record(s) selected.
Its MySQL counterpart being
mysql> describe mytable;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| name | varchar(512) | NO | | NULL | |
+-------+--------------+------+-----+---------+----------------+
2 rows in set (0.01 sec)
Let's assume the DB2 and MySQL databases are called mydb.
Now, if I do
mysqldump -uroot mydb mytable --no-create-info --extended-insert=FALSE --complete-insert | # mysldump, with options (see below): # do not output table create statement # one insert statement per record# ouput table column names
sed -n -e '/^INSERT/p' | # only keep lines beginning with "INSERT"
sed 's/`/"/g' | # replace ` with "
sed 's/;$//g' | # remove `;` at end of insert query
sed "s/\\\'/''/g" # replace `\'` with `''` , see http://stackoverflow.com/questions/2442205/how-does-one-escape-an-apostrophe-in-db2-sql and http://stackoverflow.com/questions/2369314/why-does-sed-require-3-backslashes-for-a-regular-backslash
, I get:
INSERT INTO "mytable" ("id", "name") VALUES (1,'record 1')
INSERT INTO "mytable" ("id", "name") VALUES (2,'record 2')
INSERT INTO "mytable" ("id", "name") VALUES (3,'record 3')
INSERT INTO "mytable" ("id", "name") VALUES (4,'record 4')
INSERT INTO "mytable" ("id", "name") VALUES (5,'" "" '' '''' \"\" ')
This ouput can be used as a DB2 query and it works well.
Any idea on how to solve this more efficiently/generally? Any other suggestions?
After having played around a bit I came with the following routine which I believe to be fairly general, robust and scalable.
1 Run the following command:
mysqldump -uroot mydb mytable --no-create-info --extended-insert=FALSE --complete-insert | # mysldump, with options (see below): # do not output table create statement # one insert statement per record# ouput table column names
sed -n -e '/^INSERT/p' | # only keep lines beginning with "INSERT"
sed 's/`/"/g' | # replace ` with "
sed -e 's/\\"/"/g' | # replace `\"` with `#` (mysql escapes double quotes)
sed "s/\\\'/''/g" > out.sql # replace `\'` with `''` , see http://stackoverflow.com/questions/2442205/how-does-one-escape-an-apostrophe-in-db2-sql and http://stackoverflow.com/questions/2369314/why-does-sed-require-3-backslashes-for-a-regular-backslash
Note: here unlike in the question ; are not being removed.
2 upload the file to DB2 server
scp out.sql user#myserver:out.sql
3 run queries from the file
db2 -tvsf /path/to/query/file/out.sql