I am using PSQL. My command line is:
$\copy (select current_timestamp) to '/home/myname/outputfile.txt'
I would like to know, How do I replace "(select current_Timestamp)" with a filename that houses that same select statement?
ex:
$\copy (My_SQL_FILE.sql) to '/home/myname/outputfile.txt'
I've tried googling but I can't seem to find the answer.
$\copy (my_Sql_file.sql) to '/home/myname/outputfile.txt'
does not work
You want to run a query stored in a file in a \copy statement, i.e. execute that query and store the output in a file? That is doable.
I've come across this use-case myself and wrote psql2csv. It takes the same arguments as psql and additionally takes a query as parameter (or through STDIN) and prints the output to STDOUT.
Thus, you could use it as follows:
$ psql2csv [CONNECTION_OPTIONS] < My_SQL_FILE.sql > /home/myname/outputfile.txt
What psql2csv will basically do is transform your query to
COPY ($YOUR_QUERY) TO STDOUT WITH (FORMAT csv, ENCODING 'UTF8', HEADER true)
and execute that through psql. Note that you can customize a bunch of things, such as headers, separator, etc.
Related
As described in the title, the issue is that psql variables, set using \set work for me except when used inside the \copy function provided by the psql client
Is there a some special syntax required to reference a psql variable inside a \copy? Or am I out of luck? And if so, is this documented anywhere?
I couldn't find this issue in StackOverflow or documented anywhere. I looked at ~20 posts but found nothing. I also checked the documentation on \copy for PostgreSQL 11 (the version of the CLI) and saw no caveats about this- I searched the page for "variable" and found nothing related to this. I also searched for "expansion" and "expand" and found nothing. So now I'm here asking for help...
The version of PostgreSQL client is 11.10 with whatever downstream patches Debian applied:
psql (PostgreSQL) 11.10 (Debian 11.10-1.pgdg100+1)
I'm pretty sure the server version has little to no relevance, but just to be thorough, the server is version 10.13 as shipped by Ubuntu:
psql (PostgreSQL) 10.13 (Ubuntu 10.13-1.pgdg16.04+1)
Reproducing
I'm aware of the difference between \copy and COPY (one being implemented as a feature in the psql client, the other being a server feature executing in the context of the server process) and for this task, what I need to use is definitely \copy
The standard query that shows I'm setting and referencing the variables correctly:
[local:/tmp]:5432 dbuser#dbdev# \set var_tname ag_test
[local:/tmp]:5432 dbuser#dbdev# \set var_cname fname
[local:/tmp]:5432 dbuser#dbdev# SELECT * from :var_tname WHERE :var_cname = 'TestVal' LIMIT 1;
fname|lname|score|nonce
TestVal|C|100|b
(1 row)
Time: 88.786 ms
The failing case(s), which seem to be failing because the variables are referenced inside of \copy- I don't see any other difference between this and the working example:
[local:/tmp]:5432 dbuser#dbdev# \set var_tname ag_test
[local:/tmp]:5432 dbuser#dbdev# \set var_cname fname
[local:/tmp]:5432 dbuser#dbdev# \copy (SELECT * from :var_tname WHERE :var_cname = 'TestVal' LIMIT 1) TO 'testvar.csv';
ERROR: syntax error at or near ":"
LINE 1: COPY ( SELECT * from :var_tname WHERE :var_cname = 'TestVal...
^
Time: 193.322 ms
Obviously, based on the error, the expansion is not happening and the query is trying to reference a table with a literal name of :var_tname
I wasn't expecting quoting to help with this, but tried it just in case- who knows, could be a bizarre exception, right? Unsurprisingly, it's of no help either:
[local:/tmp]:5432 dbuser#dbdev# \copy (SELECT * from :'var_tname' WHERE :var_cname = 'TestVal' LIMIT 1) TO 'testvar.csv';
ERROR: syntax error at or near ":"
LINE 1: COPY ( SELECT * from : 'var_tname' WHERE :var_cname = 'Test...
^
Time: 152.407 ms
[local:/tmp]:5432 dbuser#dbdev# \set var_tname 'ag_test'
[local:/tmp]:5432 dbuser#dbdev# \copy (SELECT * from :var_tname WHERE :var_cname = 'TestVal' LIMIT 1) TO 'testvar.csv';
ERROR: syntax error at or near ":"
LINE 1: COPY ( SELECT * from :var_tname WHERE :var_cname = 'TestVal...
^
Time: 153.001 ms
[local:/tmp]:5432 dbuser#dbdev# \copy (SELECT * from :'var_tname' WHERE :var_cname = 'TestVal' LIMIT 1) TO 'testvar.csv';
ERROR: syntax error at or near ":"
LINE 1: COPY ( SELECT * from : 'var_tname' WHERE :var_cname = 'Test...
^
Time: 153.459 ms
I also tried setting the variables with single quotes (which is probably a best practice anyway) but this made no difference:
[local:/tmp]:5432 dbuser#dbdev# \set var_tname 'ag_test'
[local:/tmp]:5432 dbuser#dbdev# \set var_cname 'fname'
... <same behavior as above> ...
Is variable expansion just not supported inside of \copy? If so, that seems like a really crummy limitation, and it doesn't seem to be documented
One last thing to add as I expect someone may ask- there is a reason I'm not implementing these as functions or stored procedures. First, my version of PostgreSQL doesn't support stored procedures at all. It also does not support transactions in functions. Even if it did, the real reason I want these queries in psql files in the application repository is because they are then very easy to read for code reviews, easy to maintain for development, and act as documentation
It's not necessary to read beyond this point unless you also have this problem and want ideas for workarounds
Beyond this I've documented a bunch of workarounds I could quickly think of- the problem can be worked around 1001 different ways. But if there's a solution to this quirky behavior that lets me stick with it, I would much rather know about it than apply any workarounds. I also added use-case information below as it's not unheard of for responses to be along the lines of "why are you doing this? Just don't use xyz feature, problem solved!". I'm hoping to not receive any of those responses :>
Thanks for anyone willing to help out!
Options for Workarounds
I have plenty of options for workarounds, but I really would like to understand why this doesn't work, if it's documented somewhere, or if there may be some special way to cause the expansion to occur when used inside \copy, to avoid needing to change this- for reasons I explain below in the Use-Case section
Here are the workarounds I've come up with...
SELECT into temporary table using the variables, \copy that fixed name table
SELECT * INTO tmp_table FROM :var_tname WHERE :var_cname = 'TestVal' LIMIT 1;
\copy (SELECT * FROM tmp_table) TO 'testvar.csv'
That works, but it's kind of clunky and seems like it shouldn't be unnecessary
Produce a TSV using \pset fieldsetp and redirect stdout to the file (clunky, may have escaping issues)
The other option would be not using \copy and piping stdout to a file after setting the delimiter to tab:
[local:/tmp]:5432 dbuser#dbdev# \set var_tname ag_test
[local:/tmp]:5432 dbuser#dbdev# \pset format unaligned
Output format is unaligned.
[local:/tmp]:5432 dbuser#dbdev# \pset fieldsep '\t'
Field separator is " ".
[local:/tmp]:5432 dbuser#dbdev# SELECT * from :var_tname LIMIT 1;
fname lname score nonce
TestVal G 500 a
(1 row)
Time: 91.596 ms
[local:/tmp]:5432 dbuser#dbdev#
This could be invoked via psql -f query.psql > /the/output/path.tsv. I haven't checked yet but I'm assuming that should produce a valid TSV file. The one thing I'm not sure of is whether it will correctly escape or quote column values that contain tabs, like \copy or COPY would
Do the expansion in a shell script and write to temporary psql file, use psql -f tmp.psql
The final workaround would be setting the variables in a shell script, and invoking using psql -c "$shellvar", or writing the shell-expanded query to a temporary .psql file and then invoking with psql -f, and deleting the temporary file
The Use-Case (and why I don't particularly like some of the workarounds)
I should probably mention the use case... I have several separate (but related) Python applications that collect, parse and process data and then load them into the database using psycopg2. Once the raw data is in the database, I delegate a bunch of heavier logic into psql files, for readability and to reduce the amount of code that needs to be maintained
The psql files are invoked at completion of the application using something like this:
for psql_file in glob.glob(os.path.expandvars("$VIRTUAL_ENV/etc/<appname>/psql-post.d/*.psql:
subprocess.call([which('psql'), '-f', psql_file])
One of the reasons I want to use variables for the table names (and some column names) is because the database is currently being refactored/rebuilt, so the table names and a few column names are going to be renamed over time. Because some of the .psql scripts are quite extensive, the table names are referenced quite a few times in them- so it just makes more sense to set them once at the top, using \set, so as each table gets changed in the database, only one change in each psql file is required. There also may be minor changes in the future that make this approach better than one where you need to search and replace 10-15 instances of various column or table names
One last workaround I don't really want to use: templating psql files from Python
I realize I could use some home-grown templating or Jinja2 directly from the Python code to dynamically generate the PSQL files from templates as well. But I much prefer to have pure psql in the files as it makes the project much more readable and editable for those who may need to perform a code review, or take over maintenance of the projects in the future. It's also easier for me to work with. Obviously there are tons of options for workarounds once we start talking about doing this from within Python using queries via psycopg2- but having the .psql files in the same relative directory of each project repository serves a very useful purpose
It seems to be a parsing issue with \copy.
UPDATE: Actually a documented behavior:
https://www.postgresql.org/docs/current/app-psql.html
\copy
...
Unlike most other meta-commands, the entire remainder of the line is always taken to be the arguments of \copy, and neither variable interpolation nor backquote expansion are performed in the arguments.
Tip
Another way to obtain the same result as \copy ... to is to use the SQL > COPY ... TO STDOUT command and terminate it with \g filename or \g >|program. Unlike \copy, this method allows the command to span multiple > lines; also, variable interpolation and backquote expansion can be used.
\set var_tname 'cell_per'
\copy (select * from :var_tname) to stdout WITH (FORMAT CSV, HEADER);
ERROR: syntax error at or near ":"
LINE 1: COPY ( select * from :var_tname ) TO STDOUT WITH (FORMAT CS...
\copy (select * from :"var_tname") to stdout WITH (FORMAT CSV, HEADER);
ERROR: syntax error at or near ":"
LINE 1: COPY ( select * from : "var_tname" ) TO STDOUT WITH (FORMAT...
--Note the added space when using the suggested method of including a variable as
--table name.
copy (select * from :var_tname) to stdout WITH (FORMAT CSV, HEADER);
copy (select * from :"var_tname") to stdout WITH (FORMAT CSV, HEADER);
--Using COPY directly works.
--So:
\o cp.csv
copy (select * from :var_tname) to stdout WITH (FORMAT CSV, HEADER);
\o
--This opens file cp.csv COPYs to it and then closes file.
-- Or per docs example and UPDATE:
copy (select * from :var_tname) to stdout WITH (FORMAT CSV, HEADER) \g cp.csv
cat cp.csv
line_id,category,cell_per,ts_insert,ts_update,user_insert,user_update,plant_type,season,short_category
5,H PREM 3.5,18,,06/02/2004 15:11:26,,postgres,herb,none,HP3
7,HERB G,1,,06/02/2004 15:11:26,,postgres,herb,none,HG
9,HERB TOP,1,,06/02/2004 15:11:26,,postgres,herb,none,HT
10,VEGGIES,1,,06/02/2004 15:11:26,,postgres,herb,annual,VG
Long story...
I am trying to geenrate a crosstab query dynamically and run it as a psql script..
To achieve this, I want the last line of the sql to generated and appended to the top portion of the sql.
The last line of the sql is like this.... "as final_result(symbol character varying,"431" numeric,"432" numeric,"433" numeric);"
Of which, the "431", "432" etc are to be generated dynamically as these are the pivot columns and they change from time to time...
So I wrote a query to output the text as follows....
psql -c "select distinct '"'||runweek||'" numeric ,' from calendar where runweek between current_runweek()-2 and current_runweek() order by 1;" -U USER -d DBNAME > /tmp/gengen.lst
While the sql provides the output, when I run it as a script, because of the special characters (', "", ) it fails.
How should I get it working? My plan was then loop through the "lst" file and build the pivot string, and append that to the top portion of the sql and execute the script... (New to postgres, does not know , dynamic sql generation and execution etc.. but I am comfortable with UNIX scripting..)
If I could somehow get the output as
("431" numeric, "432" numeric....etc) in a single step, if there is a recommendation to achieve this, it will be greatly appreciated.....
Since you're using double quotes around the argument, double quotes inside the argument must be escaped with a backslash:
psql -c "select distinct '\"'||runweek||'\" numeric ,' from calendar where runweek between current_runweek()-2 and current_runweek() order by 1;"
Heredoc can also be used instead of -c. It accepts multi-line formatting so that makes the whole thing more readable.
(psql [arguments] <<EOF
select distinct '"'||runweek||'" numeric ,'
from calendar
where runweek between current_runweek()-2 and current_runweek()
order by 1;
EOF
) > output
By using quote_ident which is specifically meant to produce a quoted identifier from a text value, you don't even need to add the double quotes. The query could be like:
select string_agg( quote_ident(runweek::text), ',' order by runweek)
from calendar
where runweek between current_runweek()-2 and current_runweek();
which also solves the problem that your original query has a stray ',' at the end, whereas this form does not.
For example, say I've got the output of:
SELECT
$text$col1, col2, col3
0,my-value,text
7,value2,string
0,also a value,fort
$text$;
Would it be possible to populate a table directly from it with the COPY command?
Sort of. You would have to strip the first two and last lines of your example in order to use the data with COPY. You could do this by using the PROGRAM keyword:
COPY table_name FROM PROGRAM 'sed -e ''1,2d;$d'' inputfile';
Which is direct in that you are doing everything from the COPY command and indirect in that you are setting up an outside program to filter your input.
There's 1 column that contains commas. When I output my query to csv, these commas break the csv format. What I've been doing to avoid this is a simple
replace(A."Sales Rep",',','')
Is there a better way of doing this so that I can actually get the commas in the final output without breaking the csv file?
Thanks!
You can use the COPY command to get PostgreSQL to build the CSV for you:
COPY -- copy data between a file and a table
Something like one of these:
copy your_table to 'filename' csv
copy your_table to 'filename' csv force quote *
copy your_table to stdout csv force quote *
copy your_table to stdout csv force quote * header
...
You have to be the super user to copy to a filename though. If you're inside psql, you can use the \copy command:
Performs a frontend (client) copy. This is an operation that runs an SQL COPY command, but instead of the server reading or writing the specified file, psql reads or writes the file and routes the data between the server and the local file system.
The syntax is pretty much the same:
\copy your_table to 'filename.csv' csv force quote * header
...
Quote the fields with "
a,this has a , in it,b
would become
a,"this has a, in it",b
and if the fields have BOTH a , and a ", double the quotes:
a,this has a " and , in it,b
becomes
a,"this has a "" and , in it",b
A while back I was looking for a way to insert values into a text field through isql
and eventually found some load command that worked out for me.
It doesn't work when I try to execute it from Perl. I get a syntax error. I have tried two separate methods and both are not working so far.
I have the SQL statement variable print out at the end of each loop cycle so I know that the syntax is correct, but just not getting across correctly.
Here's the latest snip of code I was testing:
foreach(#files)
{
$STMT = <<EOF;
load from $_ insert into some_table
EOF
$sth = $db1->prepare($STMT);
$sth->execute;
}
#files is an array whose elements are a full path/location of a pipe-delimited text file (ex. /home/xx/xx/xx/something.txt)
The number of columns in the table match the number of fields in the text file and the type-checking is fine (I've loaded test files manually without fail)
The error I get back is:
DBD::Informix::db prepare failed: SQL: -201: A syntax error has occurred.
Any idea what might be causing this?
EDIT to RET's & Petr's answers
$STMT = "'LOAD FROM $_ INSERT INTO table'";
system("echo $STMT | isql $db")
I had to change it to this, because the die command would force an unnatural death and the statement had to be wrapped in single quotes.
Petr is exactly right, the LOAD statement is an ISQL or DB-Access extension, so you can't execute it through DBI. If you have a look at the manual, you'll see it is also invalid syntax for SPL, ESQL/C and so on.
It's not clear whether you have to use perl to execute the script, or perl is just a convenient way of generating the SQL.
If the former, and you want a pure-perl method, you have to prepare an INSERT statement (there's just one table involved by the look of it?), and slurp through the file, using split to break it up into columns and executing the prepared insert.
Otherwise, you can generate the SQL using perl and execute it through DB-Access, either directly with system or by wrapping both in either a shell script or DOS batch file.
System call version
foreach (#files) {
my $stmt = "LOAD FROM $_ INSERT INTO table;\n";
system("echo $stmt | dbaccess $database")
|| die "Statement $stmt failed: $!\n";
}
In a batch script version, you could write all the SQL into a single script, ie:
perl -e 'while(#ARGV){shift; print "LOAD FROM '$_' INSERT INTO table;\n"}' file1 [ file2 ... ] > loadfiles.sql
isql database loadfiles.sql
NB, the comment about quotes on the filename is only relevant if the filename contains spaces or metacharacters, the usual issue.
Also, one key difference in behaviour between isql and dbaccess is that when executed in this manner, dbaccess does not stop on error, but isql will. To make dbaccess stop processing on error, set DBACCNOIGN=1 in the environment.
Hope that's helpful.
This is because your query is not SQL query, it is an isql command that tells isql to parse the input file and generate INSERT statements.
If you think about it, the server can be on a completely different machine and has no idea what file are you talking about and how to access it.
So you basically have two options:
call isql and pipe the LOAD command to it - very ugly
parse the file yourself and generate the INSERT statements
Please note that the file Notes/load.unload is distributed with DBD::Informix and contains guidelines on how to handle UNLOAD operations using Perl, DBI and DBD::Informix. Somewhat to my chagrin, I see that it says "T.B.D." (more or less) for the LOAD section.
As other people have stated, the LOAD and UNLOAD statements are faked by various client-side tools to look like SQL statements, but the Informix server does not support them itself, mainly because of the issue with getting the file from a client machine (perhaps a PC) to the server machine (perhaps a Solaris machine).
To simulate the LOAD statement, you would need to analyze the INSERT INTO Table part. If it lists columns (INSERT INTO Table(Col03, Col05, Col09)), then you can expect three values in the load data file, and they go into those three columns. You would prepare a statement 'SELECT Col03, Col05, Col09 FROM Table' to get the types of the columns. Otherwise, you need to prepare a statement 'SELECT * FROM Table' to get the complete list of columns (and their types). Given the column names and the number of columns, you can create and prepare a suitable insert statement: 'INSERT INTO Table(Col03, Col05, Col09) VALUES(?,?,?)' or 'INSERT INTO Table VALUES(?,?,?,?,?,?,?,?,?)'. You could (arguably should) include column names in the second one.
With that ready, you now have parse the unloaded data. There is a document available in the SQLCMD program available from the IIUG Software Archive (which has been around a lot longer than Microsoft's upstart program of the same name). That describes the UNLOAD format in considerable detail. Perl has the ability to handle anything Informix uses - witness the UNLOAD information in the load.unload file distributed with DBD::Informix.
A quick bit of Googling showed that the syntax for load puts quote marks around the file name. What if you change your statement to be:
load from '$_' insert into some_table
Since your statement is not using place holders, you have to put the quotes in yourself, as opposed to using the DBI quoting functionality.