How to copy with statement result to local in postgresql - postgresql

I have following with statement and copy command
with output01 as
(select * from (
select name,
case
when column1 is not null and lower(column1) in ('point1','point2','point3','point4') then 3456
else null end column1Desc,
case
when column2 is not null and lower(column2) in ('point1','point2','point3','point4') then 2456
else null end column2Desc,
column3, column4),
output02 as
(select * from (
select name,
case
when column1 is not null and lower(column1) in ('point1','point2','point3','point4') then 3456
else null end column1Desc,
case
when column2 is not null and lower(column2) in ('point1','point2','point3','point4') then 2456
else null end column2Desc,
column3, column4),
output3 as (SELECT * FROM output01 UNION ALL SELECT * FROM output02)
\copy (select * from output3) to '/usr/share/output.csv' with CSV ENCODING 'UTF-8' DELIMITER ',' HEADER;
I am getting following ERROR
ERROR: relation "tab3" does not exist

All psql backslash commands need to be written on a single line, so you can't have a multi-line query together with \copy. The only workaround is to create a (temporary) view with that query, then use that in the \copy command.
Something along the lines:
create temporary view data_to_export
as
with cte as (..)
select *
from cte
;
\copy (select * data_to_export) to ...

You are getting this error because you are running your CTE query and copy command in different statements. Considering your with query is working fine, you should write your copy statement like below:
\copy (WITH tab1 as (Your SQL statement),
tab2 as ( SELECT ... FROM tab1 WHERE your filter),
tab3 as ( SELECT ... FROM tab2 WHERE your filter)
SELECT * FROM tab3) to '/usr/share/results.csv' with CSV ENCODING 'UTF-8' DELIMITER ',' HEADER;

Related

DB2 - String operations

Can we check a string value with comma separated exists in another comma separated string in DB2?
I have say two string like below
Case-1:
String-1:'ABC,XYZ,PQR'
Srting-2:'MNO,PQR'
I want to have any function in DB2 which should return 1 in the above case as PQR from String-2 exists in String-1. The value should match any string within comma or after or before comma.
Case-2:
String-1:'ABC,XYZ,PQR'
Srting-2:'MNO,P QR'
But for the above case the same function should return 0. As there is space after "P"
I also want to have a similar function which should return the concatenated string but removes duplicates.
Case-1:
String-1:'ABC,XYZ,PQR'
Srting-2:'MNO,PQR'
Output should be 'ABC,XYZ,PQR,MNO' .
The order in comma separated values doesn't matter in this case; but PQR should not be repeated twice.
Case-2:
String-1:'ABC,XYZ,PQR'
Srting-2:'MNO,P QR'
In this case output should be 'ABC,XYZ,PQR,MNO,P QR' .
I am trying to achieve this using SQL or any function in DB2.
There is another case where I want to remove matching part of the string
Case-3:
String-1:'ABC,XYZ,PQR'
Srting-2:'MNO,PQR'
Output should be 'ABC,XYZ,MNO'
Obviously String-1 and String-2 can have any number of values separated by comma.
Try this:
CREATE FUNCTION SPLITTER (P_STR VARCHAR(4000), P_DELIM VARCHAR(10))
RETURNS TABLE (SEQ INT, TOKEN VARCHAR(50))
RETURN
SELECT TOK.SEQ, TOK.TOKEN
FROM XMLTABLE
(
'for $id in tokenize($s, $t) return <i>{string($id)}</i>' PASSING P_STR AS "s", P_DELIM AS "t"
COLUMNS
SEQ FOR ORDINALITY
, TOKEN VARCHAR(50) PATH '.'
) TOK;
SELECT
CASE
WHEN EXISTS
(
SELECT 1
FROM TABLE (SPLITTER ('ABC,XYZ,PQR', ',')) A
JOIN TABLE (SPLITTER ('MNO,PQR', ',')) B ON B.TOKEN = A.TOKEN
)
THEN 1
ELSE 0
END
FROM SYSIBM.SYSDUMMY1;
SELECT LISTAGG (TOKEN, ',')
FROM
(
SELECT TOKEN
FROM TABLE (SPLITTER ('ABC,XYZ,PQR', ','))
UNION
SELECT TOKEN
FROM TABLE (SPLITTER ('MNO,PQR', ','))
);
SELECT LISTAGG (TOKEN, ',')
FROM
(
SELECT DISTINCT COALESCE (A.TOKEN, B.TOKEN) AS TOKEN
FROM TABLE (SPLITTER ('ABC,XYZ,PQR', ',')) A
FULL JOIN TABLE (SPLITTER ('MNO,PQR', ',')) B ON B.TOKEN = A.TOKEN
WHERE A.TOKEN IS NULL OR B.TOKEN IS NULL
);
dbfiddle link.

Execute result of a PostgreSQL query again to get the final result set?

I'm looking for a way to get the list of all json attributes across all my PostgreSQL tables dynamically.
I have Query 1 which would generate a list of sql statements, and then run that sql statements to get the final output all in one go (like the dynamic SQL concept in SQL server).
Query 1 looks like this :
create temporary table test (ordr int, field varchar(1000));
-- Step 1 Create temp table to insert all table/col/json attrbute info
insert into test(ordr,field)
select 0 ordr,'create temporary table temp_table
( table_schema varchar(200)
,table_name varchar(200)
,column_name varchar(200)
,json_attribute varchar(200)
,data_type varchar(50)
);'
union
-- Non json type columns
select 1 ordr, 'insert into temp_table(table_name, column_name,data_type,json_attribute)'
union
-- Json columns with data like json object
select
3 ordr,
concat('select distinct ''', t.table_name, ''' tbl, ''', c.column_name, ''' col, ''' , c.data_type,''' data_type, '
,'jsonb_object_keys(', c.column_name, ') json_attribute', ' from ', t.table_name,
' where jsonb_typeof(' , c.column_name, ') = ''object'' union') AS field
from information_schema.tables t
join information_schema.columns c on c.table_name = t.table_name
where t.table_schema not in ('information_schema', 'pg_catalog')
--and table_type = 'BASE TABLE'
and c.data_type ='jsonb';
--final sql statements to build temp table
--copy all the column "txt" to a separate window and execute it, it will create a temp table "temp_table" which will have all tables/cols/json attributes
select ordr
,(case when t.ordr = (select max(t2.ordr) from test t2) then replace(field,'union','') else field end) txt
from test t
union
select 9999, ';select * from temp_table;'
order by 1 ;
Query 1 output : This is a list of sql statements
I'm looking for a way to run the Query 1 & Query 1 output which would get me the final output all in one go.
Any lead or guidance will be really appreciated.

Working around the 1600 column max in Postgres during csv import

I am trying to import a massive csv file (nearly 10000 columns) into a postgres database. Normally to import a file I would copy it to a temporary table and then use that temp table to place everything in its correct place. However postgres has a max columns of 1600 so the csv doesn't import. I don't have any control over the size of the csv or how it is formatted so I need to work with it.
Is there either a way to increase this value for temporary tables, or to use the copy command to parse into multiple temporary tables? I would also be ok using another way of importing the file if you have any suggestions.
Any advice? I am currently using the Copy command to import the csv:
COPY INTAKE
FROM 'file location'
CSV HEADER;
Thank you for your help!
Import it all as a single column, as described here:
https://stackoverflow.com/a/60120879/16361
Once you have that, you can put together a query that will generate SQL to extract the fields that you want in the way that you want. A small example with 5 columns and a max table size of 2 columns (IRL, you'd want to max_cols_per_table in the code block following this one to a large number, like 1kish, so you're creating 10 tables instead of 5000).
$ cat /tmp/fivewide.csv
c1,c2,c3,c4,c5
r1c1,r1c2,r1c3,r1c4,r1c5
r2c1,r2c2,r2c3,r2c4,r2c5
$ psql -X testdb
psql (12.3 (Debian 12.3-1), server 10.5 (Debian 10.5-1+build4))
Type "help" for help.
testdb=# create table my_import_table(data text);
CREATE TABLE
testdb=# \copy my_import_table from /tmp/fivewide.csv csv delimiter e'\x01' quote e'\x02'
COPY 3
testdb=# select * from my_import_table;
data
--------------------------
c1,c2,c3,c4,c5
r1c1,r1c2,r1c3,r1c4,r1c5
r2c1,r2c2,r2c3,r2c4,r2c5
(3 rows)
Generating the CREATE TABLE statments so it's split into actual columns (this will break if there are commas in the values; I'm not going to implement a full CSV parser in SQL :)
testdb=# with
settings as (select 2 as max_cols_per_table, 'my_import_table' as import_table, 'data' as column_name),
computed_settings1 as (select array_length(string_to_array(d, ','), 1) as num_cols from t limit 1),
computed_settings2 as (select (ceil((select num_cols::float from computed_settings1) / (select max_cols_per_table from settings)))::integer as num_tables),
columns_exprs as (select i, '(string_to_array('||(select column_name from settings)||$$, ','))[$$||i||'] AS col'||i as cexpr from generate_series(1, (select num_cols from computed_settings1)) as i),
column_exprs_by_table as (select t, cexpr from generate_series(1, (select num_tables from computed_settings2)) as t join columns_exprs ce
on i>((t-1)*(select max_cols_per_table from settings))
AND i<=(t*(select max_cols_per_table from settings))
),
create_table_stmts as (select 'create table t_'||t||' AS SELECT '||string_agg(cexpr, ', ')||' FROM '||(select import_table from settings)||';' from column_exprs_by_table group by t)
select * from create_table_stmts;
?column?
-----------------------------------------------------------------------------------------------------------------------------------
create table t_3 AS SELECT (string_to_array(data, ','))[5] AS col5 FROM my_import_table;
create table t_2 AS SELECT (string_to_array(data, ','))[3] AS col3, (string_to_array(data, ','))[4] AS col4 FROM my_import_table;
create table t_1 AS SELECT (string_to_array(data, ','))[1] AS col1, (string_to_array(data, ','))[2] AS col2 FROM my_import_table;
(3 rows)
Executing this DDL with DO and inspecting the results:
testdb=# DO $outer$
DECLARE
stmt text;
BEGIN
FOR stmt IN
with
settings as (select 2 as max_cols_per_table, 'my_import_table' as import_table, 'data' as column_name),
computed_settings1 as (select array_length(string_to_array(d, ','), 1) as num_cols from t limit 1),
computed_settings2 as (select (ceil((select num_cols::float from computed_settings1) / (select max_cols_per_table from settings)))::integer as num_tables),
columns_exprs as (select i, '(string_to_array('||(select column_name from settings)||$$, ','))[$$||i||'] AS col'||i as cexpr from generate_series(1, (select num_cols from computed_settings1)) as i),
column_exprs_by_table as (select t, cexpr from generate_series(1, (select num_tables from computed_settings2)) as t join columns_exprs ce
on i>((t-1)*(select max_cols_per_table from settings))
AND i<=(t*(select max_cols_per_table from settings))
),
create_table_stmts as (select 'create table t_'||t||' AS SELECT '||string_agg(cexpr, ', ')||' FROM '||(select import_table from settings)||';' from column_exprs_by_table group by t)
select * from create_table_stmts
LOOP
EXECUTE stmt;
END LOOP;
END;
$outer$;
DO
testdb=# select * from t_1;
col1 | col2
------+------
c1 | c2
r1c1 | r1c2
r2c1 | r2c2
(3 rows)
testdb=# select * from t_2;
col3 | col4
------+------
c3 | c4
r1c3 | r1c4
r2c3 | r2c4
(3 rows)
testdb=# select * from t_3;
col5
------
c5
r1c5
r2c5
(3 rows)
Further improvements, perhaps to be smart about using row 1 as column names, or to always include some column that would be useful for joining the tables, are left as an exercise to the reader.

PostgreSQL \copy from ... with CSV null as '' still imports empty strings instead of NULLs

I thought I knew PostgreSQL pretty well and have done this many times before, but I have here
\copy MyTable FROM ... WITH CSV NULL AS ''
and despite that a row such as:
"FooBar","","","1","0","","29505;29505",""
will not get its second column imported as NULL but as empty string!
SELECT count(1) FROM MyTable WHERE col2 IS NULL
== 0
SELECT count(1) FROM MyTable WHERE col2 = ''
>= 1
What am I doing wrong? Such a simple thing!
Here is a completely self-sufficient test to prove this:
CREATE TABLE Test(col1 text, col2 int);
\copy Test FROM PROGRAM 'printf '',\na,2\n''' WITH DELIMITER ',' NULL ''
SELECT (SELECT count(1) FROM Test WHERE col1 IS NULL) AS col1null,
(SELECT count(1) FROM Test WHERE col2 IS NULL) AS col2null,
(SELECT count(1) FROM Test WHERE col1 = '') AS col1empty;
TRUNCATE Test;
\copy Test FROM PROGRAM 'printf ''"",""\n"a","2"\n''' WITH CSV NULL ''
\copy Test FROM PROGRAM 'printf ''"","0"\n"a","2"\n''' WITH CSV NULL ''
SELECT (SELECT count(1) FROM Test WHERE col1 IS NULL) AS col1null,
(SELECT count(1) FROM Test WHERE col2 IS NULL) AS col2null,
(SELECT count(1) FROM Test WHERE col1 = '') AS col1empty;
DROP TABLE Test;
And here is how it runs:
foo=# CREATE TABLE Test(col1 text, col2 int);
SELECT (SELECT count(1) FROM Test WHERE col1 IS NULL) AS col1null,
(SELECT count(1) FROM Test WHERE col2 IS NULL) AS col2null,
(SELECT count(1) FROM Test WHERE col1 = '') AS col1empty;
TRUNCATE Test;
\copy Test FROM PROGRAM 'printf ''"",""\n"a","2"\n''' WITH CSV NULL ''
\copy Test FROM PROGRAM 'printf ''"","0"\n"a","2"\n''' WITH CSV NULL ''
SELECT (SELECT count(1) FROM Test WHERE col1 IS NULL) AS col1null,
(SELECT count(1) FROM Test WHERE col2 IS NULL) AS col2null,
(SELECT count(1) FROM Test WHERE col1 = '') AS col1empty;
DROP TABLE Test;CREATE TABLE
foo=# \copy Test FROM PROGRAM 'printf '',\na,2\n''' WITH DELIMITER ',' NULL ''
COPY 2
foo=# SELECT (SELECT count(1) FROM Test WHERE col1 IS NULL) AS col1null,
foo-# (SELECT count(1) FROM Test WHERE col2 IS NULL) AS col2null,
foo-# (SELECT count(1) FROM Test WHERE col1 = '') AS col1empty;
col1null | col2null | col1empty
----------+----------+-----------
1 | 1 | 0
(1 row)
foo=# TRUNCATE Test;
TRUNCATE TABLE
foo=# \copy Test FROM PROGRAM 'printf ''"",""\n"a","2"\n''' WITH CSV NULL ''
ERROR: invalid input syntax for integer: ""
CONTEXT: COPY test, line 1, column col2: ""
foo=# \copy Test FROM PROGRAM 'printf ''"","0"\n"a","2"\n''' WITH CSV NULL ''
COPY 2
foo=# SELECT (SELECT count(1) FROM Test WHERE col1 IS NULL) AS col1null,
foo-# (SELECT count(1) FROM Test WHERE col2 IS NULL) AS col2null,
foo-# (SELECT count(1) FROM Test WHERE col1 = '') AS col1empty;
col1null | col2null | col1empty
----------+----------+-----------
0 | 0 | 1
(1 row)
foo=# DROP TABLE Test;
DROP TABLE
So what can I do to get "" CSV as in text columns interpreted as NULL?
I got it now
\copy Test FROM PROGRAM 'printf ''"",""\n"a","2"\n''' WITH(FORMAT CSV, DELIMITER ',', NULL '', FORCE_NULL(col1,col2))
Turns out the COPY syntax is very lax, but if you really want to make it work, you must use the WITH with parentheses and do it all correctly. Weird.

For xml - similar function in postgresql [duplicate]

Let say you have a SELECT id from table query (the real case is a complex query) that does return you several results.
The problem is how to get all id return in a single row, comma separated?
SELECT string_agg(id::text, ',') FROM table
Requires PostgreSQL 9.0 but that's not a problem.
You can use the array() and array_to_string() functions togetter with your query.
With SELECT array( SELECT id FROM table ); you will get a result like: {1,2,3,4,5,6}
Then, if you wish to remove the {} signs, you can just use the array_to_string() function and use comma as separator, so: SELECT array_to_string( array( SELECT id FROM table ), ',' ) will get a result like: 1,2,3,4,5,6
You can generate a CSV from any SQL query using psql:
$ psql
> \o myfile.csv
> \f ','
> \a
> SELECT col1 AS column1, col2 AS column2 ... FROM ...
The resulting myfile.csv will have the SQL resultset column names as CSV column headers, and the query tuples as CSV rows.
h/t http://pookey.co.uk/wordpress/archives/51-outputting-from-postgres-to-csv
use array_to_string() & array() function for the same.
select array_to_string(array(select column_name from table_name where id=5), ', ');
Use this below query it will work and gives the exact result.
SELECT array_to_string(array_agg(id), ',') FROM table
Output : {1,2,3,4,5}
SELECT array_agg(id, ',') FROM table
{1,2,3,4}
I am using Postgres 11 and EntityFramework is fetching it as array of integers.