I have a users table that has an id, first_name, and last_name.
I also have a messages table that has a user_id, and text.
How do I export a user's messages to a CSV file for each user?
I can do it for one user:
COPY (SELECT text FROM messages WHERE user_id = "my user id") to "Firstname_Lastname.csv" DELIMITER ',' CSV HEADER;
but postgres doesn't seem to have a for loop or anything? I did some googling and stumbled into LATERAL but could not get that to work...
Using Adrian's link and a lot of reading and cursing at my screen I hacked something together:
CREATE OR REPLACE FUNCTION export()
RETURNS integer
LANGUAGE plpgsql
AS $$
DECLARE
myuser RECORD;
filename TEXT;
BEGIN
FOR myuser IN SELECT id,first_name,last_name FROM users LOOP
RAISE NOTICE 'user %', myuser;
EXECUTE $$SELECT '/home/me/' || $1 || '_' || $2 || '.csv'$$ INTO filename USING myuser.first_name, myuser.last_name;
RAISE NOTICE 'filename %', filename;
EXECUTE $$COPY (SELECT text FROM messages WHERE user_id = '$$ || myuser.id || $$') TO '$$ || filename || $$' DELIMITER ',' CSV HEADER$$;
END LOOP;
RETURN 1;
END;
$$
It's quite a mess and is not idiomatic at all I'm sure. But it worked for me.
Related
I am pretty fresh to PostgreSQL, so please be kind.
I am pretty sure that my problem is that I am mixing plain and dynamic SQL. I have read the relevant documentation but I am not experienced enough to see where I have gone wrong (hoping that my issue is not something more fundamental).
Currently the script is failing with a Query execution error:
SQL Error [42601]: ERROR: syntax error at or near "CREATE"
I intend to use this function to unpivot >9,000 tables (for analysis purposes); fortunately all tables have the same structure.
CREATE OR REPLACE FUNCTION all_schemaTables_unpivot(_schemaName text, _tableName text)
RETURNS void AS
$BODY$
DECLARE
_tbl record;
BEGIN
FOR _tbl IN
SELECT
quote_ident(schemaname) || '.' || quote_ident(tablename) AS fName,
quote_ident(tablename) AS tName
FROM pg_tables
WHERE schemaname = _schemaName
AND tablename LIKE _tableName
LOOP
EXECUTE 'CREATE TABLE ' || _tbl.tName || '_up AS
SELECT region_id, key AS sequential_id, value
FROM (SELECT row_to_json(t.*) AS line, region_id
FROM ' || _tbl.tName || ' AS t) AS r
JOIN LATERAL json_each_text(r.line) ON (key <> "region_id")';
END LOOP;
END;
$BODY$ LANGUAGE plpgsql;
Thanks in advance.
To get the script to work I just needed to fix two things:
I originally did not set appropriate spaces around the query concatenating (thanks #KaushikNayak)
I incorrectly set the quoting values around my dynamic variables (see the 'quote_ident(_tbl.newTableName)' addition. People more expert in PostgreSQL than I will surely have a much cleaner approach - but at least it works!
Moral of the story, sometimes the fix is staring you in the face, but you have been staring at the script for too long! Leave it for awhile and the answer becomes clear.
CREATE OR REPLACE FUNCTION
all_schemaTables_unpivot(_schemaName text, _tableName text)
RETURNS void AS
$BODY$
DECLARE
_tbl record;
BEGIN
FOR _tbl IN
SELECT
quote_ident(schemaname) || '.' || quote_ident(tablename) AS fullNamePath,
quote_ident(tablename) || '_up' AS newTableName
FROM pg_tables
WHERE schemaname = _schemaName
AND tablename LIKE _tableName
LOOP
EXECUTE 'CREATE TABLE '|| quote_ident(_tbl.newTableName) ||' AS
SELECT region_id, key AS sequential_id, value
FROM (SELECT row_to_json(t.*) AS line, region_id
FROM '|| _tbl.fullNamePath ||' AS t) AS r
JOIN LATERAL json_each_text(r.line) ON (key <> "region_id");';
END LOOP;
END;
$BODY$ LANGUAGE plpgsql;
My company is going to start generating documents using data from our database and I am designing the function that will spit out the document text. These documents will need to contain data taken from multiple tables, with hundreds of columns and invariably some records will be missing data.
I am trying to make a function that will take null fields and replace them with a little error message that makes it clear to the end user that a piece of data is missing. Since the end user is totally unfamiliar with the backend, I want these messages to reference something intelligible to them.
My solution is pretty simple yet I for the life of me can't get it to work. The record identifier, table name are set as parameters in the function. The function then loops through names for each of the columns in the specified table, building a query that contains a bunch of case statements. Once the loop is complete, the identifier is appended and then the query is executed, returning the results to the calling function.
Despite reading around quite a bit, the best I can is a single column/row containing all the results - not useful to me at all, because I need to be able to easily reference specific pieces of data in the parent query. I am a beginner with Postgres and the documentation is too complex for me to understand, any help would be appreciated.
-- Function: data_handler(text, text)
-- DROP FUNCTION data_handler(text, text);
CREATE OR REPLACE FUNCTION data_handler(target_uri text, target_table TEXT)
RETURNS SETOF record AS
$BODY$
DECLARE
c text;
strSQL text;
site_only text;
result record;
BEGIN
--We need the schema for strSQL but the loop needs just the table name.
site_only = split_part(target_table, '.', 2);
FOR c IN
SELECT column_name
FROM information_schema.columns
WHERE table_name = site_only
LOOP
strSQL = concat(strSQL, chr(10), '(SELECT CASE WHEN ', c::text, '::text IS NULL THEN concat(', chr(39), '<Error:', chr(39), ', (SELECT lkp_value FROM alb_cr.lkp_field_values WHERE column_name = ', chr(39), c::text, chr(39), ')::text, ', chr(39), ' value not found>', chr(39), ')::text ELSE ',
c::text, '::text END AS ', c::text, '_convert) AS ', c::text, ',');
END LOOP;
strSQL = LEFT(strSQL, character_length(strSQL) - 1);
strSQL = concat('SELECT ', strSQL, ' FROM ', target_table, ' WHERE nm_site_id = ', chr(39), target_uri, chr(39));
RETURN QUERY EXECUTE strSQL;
RAISE NOTICE 'strSQL: %', strSQL;
--RETURN strSQL;
--RETURN QUERY EXECUTE format('SELECT ' || strSQL || 'FROM %s WHERE nm_site_id = $1', pg_typeof(target_table)) USING target_uri;
END
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION data_handler(text, text)
OWNER TO inti;
You could create views for that as well, in the following example on a schema nullsbegone:
-- create the schema to hold the views
create schema if not exists nullsbegone;
-- create a function to create the views (any and all that you might need)
create or replace function nullsbegone.f_make_view_of(p_tablename text) returns void as $f$
begin
execute ($$
create or replace view nullsbegone.$$||(select relname from pg_class where oid = $1::regclass)||$$
returns void as
select $$||array_to_string(array(
select case when not attnotnull then 'COALESCE('||quote_ident(attname)||$$::text, (SELECT '<Error:'''||lkp_value||''' value not found>' FROM alb_cr.lkp_field_values
WHERE column_name = $$||quote_literal(attname)||$$)) AS $$
else '' end || quote_ident(attname)
from pg_attribute
where attrelid = $1::regclass and attnum > 0 order by attnum
), E', \n')||$$
from $$||$1);
end;$f$ language plpgsql;
-- create the view based on a given table
select nullsbegone.f_make_view_of('yourschema.yourtable');
-- select from your view as if you were selecting from the actual table
select * from nullsbegone.yourtable
where nm_site_id = 'yoursite';
I have a simple table "id, name, content" and I would like to export all the records in files named "id_name.txt" with content from the "content" column (text type). This will create as many files as necessary. How to do that in psql?
I was thinking something like this, but parsed does not like "arow.id" and "TO filename" syntax.
do $$
declare
arow record;
filename varchar;
begin
for arow in
select id, name, template from config_templates
loop
filename := '/tmp/' || arow.name || '.txt';
COPY (select template from config_templates where id = arow.id) TO filename (FORMAT CSV);
end loop;
end;
$$;
May be it is late for you but I found the solution in your question, so wanted to share my solution here. Your looped solution didn't work because COPY command waits filename as a string constant. My solution is using dynamic query with EXECUTE. Please pay attention to double single quotes in EXECUTE params.
do $$
declare
arow record;
filename varchar;
begin
for arow in
select id, name, template from config_templates
loop
filename := '/tmp/' || arow.name::character varying(100) || '.txt';
EXECUTE format('COPY (select template from config_templates where id = ''%s'' ) TO ''%s'' (FORMAT CSV)', arow.id::character varying(100), filename);
end loop;
end;
$$;
Take psql console and type
\COPY table_name(column_name) TO 'path_of_text_file';
So my problem is simple. I have a schema prod with many tables, and another one log with the exact same tables and structure (primary keys change that's it).
When I do UPDATE or DELETE in the schema prod, I want to record old data in the log schema.
I have the following function called after a update or delete:
CREATE FUNCTION prod.log_data() RETURNS trigger
LANGUAGE plpgsql AS $$
DECLARE
v RECORD;
column_names text;
value_names text;
BEGIN
-- get column names of current table and store the list in a text var
column_names = '';
value_names = '';
FOR v IN SELECT * FROM information_schema.columns WHERE table_name = quote_ident(TG_TABLE_NAME) AND table_schema = quote_ident(TG_TABLE_SCHEMA) LOOP
column_names = column_names || ',' || v.column_name;
value_names = value_names || ',$1.' || v.column_name;
END LOOP;
-- remove first char ','
column_names = substring( column_names FROM 2);
value_names = substring( value_names FROM 2);
-- execute the insert into log schema
EXECUTE 'INSERT INTO log.' || TG_TABLE_NAME || ' ( ' || column_names || ' ) VALUES ( ' || value_names || ' )' USING OLD;
RETURN NULL; -- no need to return, it is executed after update
END;$$;
The annoying part is that I have to get column names from information_schema for each row.
I would rather use this:
EXECUTE 'INSERT INTO log.' || TG_TABLE_NAME || ' SELECT ' || OLD;
But some values can be NULL so this will execute:
INSERT INTO log.user SELECT 2,,,"2015-10-28 13:52:44.785947"
instead of
INSERT INTO log.user SELECT 2,NULL,NULL,"2015-10-28 13:52:44.785947"
Any idea to convert ",," to ",NULL,"?
Thanks
-Quentin
First of all I must say that in my opinion using PostgreSQL system tables (like information_schema) is the proper way for such a usecase. Especially that you must write it once: you create the function prod.log_data() and your done. Moreover it may be dangerous to use OLD in that context (just like *) as always because of not specified elements order.
But,
to answer your exact question the only way I know is to do some operations on OLD. Just observe that you cast OLD to text by doing concatenation ... ' SELECT ' || OLD. The default casting create that ugly double-commas. So, next you can play with that text. In the end I propose:
DECLARE
tmp TEXT
...
BEGIN
...
/*to make OLD -> text like (2,,3,4,,)*/
SELECT '' || OLD INTO tmp; /*step 1*/
/*take care of commas at the begining and end: '(,' ',)'*/
tmp := replace(replace(tmp, '(,', '(NULL,'), ',)', ',NULL)'); /*step 2*/
/* replace rest of commas to commas with NULL between them */
SELECT array_to_string(string_to_array(tmp, ',', ''), ',', 'NULL') INTO tmp; /*step 3*/
/* Now we can do EXECUTE*/
EXECUTE 'INSERT INTO log.' || TG_TABLE_NAME || ' SELECT ' || tmp;
Of course you can do steps 1-3 in one big step
SELECT array_to_string(string_to_array(replace(replace('' || NEW, '(,', '(NULL,'), ',)', ',NULL)'), ',', ''), ',', 'NULL') INTO tmp;
In my opinion this approach isn't any better from using information_schema, but it's your call.
For some fancy database maintenance for my developer database I'd like to be able to use queries to generate commands to alter the database. The thing is: I'm a complete greenhorn to PostgreSQL. I've made my attempt but have failed colorfully.
So in the end, I would like to have a table with a single column and each row would be a command (or group of commands, depending on the case) that I would think would look something like this...
DO $$
DECLARE
command_entry RECORD;
BEGIN
FOR command_entry IN SELECT * FROM list_of_commands
LOOP
EXECUTE command_entry;
END LOOP;
END;
$$;
Where the table list_of_commands could be populated with something like the following (which in this example would remove all tables from the public schema)...
CREATE TEMP TABLE list_of_commands AS
SELECT 'drop table if exists "' || tablename || '" cascade;'
FROM pg_tables
WHERE schemaname = 'public';
However, with this I get the following error...
ERROR: syntax error at or near ""drop table if exists ""dummy_table"" cascade;""
LINE 1: ("drop table if exists ""dummy_table"" cascade;")
I assume this is a matter of escaping characters, but I'm not entirely sure how to fit that into either A) the population of the table or B) the execution of each row. Does anyone know what I could do to achieve the desired result?
The command_entry variable is of type record while the EXECUTE command expects a string. What is apparently happening is that PostgreSQL turns the record into a double-quoted string, but that messes up your command. Also, your temp table does not use a column name, making things a bit awkward to work with (the column name becomes ?column?), so change both as follows:
CREATE TEMP TABLE list_of_commands AS
SELECT 'drop table if exists public.' || quote_ident(tablename) || ' cascade' AS cmd
FROM pg_tables
WHERE schemaname = 'public';
DO $$
DECLARE
command_entry varchar;
BEGIN
FOR command_entry IN SELECT cmd FROM list_of_commands
LOOP
EXECUTE command_entry;
END LOOP;
END;
$$;
But seeing that you do all of this at session level (temp table, anonymous code block), why not write a stored procedure that performs all of this housekeeping when you are ready to do spring cleaning?
CREATE FUNCTION cleanup() RETURNS void AS $$
BEGIN
FOR tbl IN SELECT tablename FROM pg_tables WHERE schemaname = 'public'
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(tbl) || ' CASCADE';
END LOOP;
-- More housekeeping jobs
END;
$$ LANGUAGE plpgsql;
This saves a lot of typing: SELECT cleanup();. Any other housekeeping jobs you have you simply add to the stored procedure.
I had trouble with Patrick's answers, so here is an updated version for postgreSQL 10.
CREATE FUNCTION droptables(sn varchar) RETURNS void AS $$
DECLARE
tbl varchar;
BEGIN
FOR tbl IN SELECT tablename FROM pg_tables WHERE schemaname = sn
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(tbl) || ' CASCADE';
END LOOP;
END;
$$ LANGUAGE plpgsql;
And then "SELECT droptables('public');".