I have a file with a list of identifying attributes, one on each line. I'd like to use these as part of a conditional select statement to psql. The same query I'm thinking of is:
SELECT * FROM mytable where mykey IN ('contents of file');
I'd like to point the IN construct to the file (which is being generated from another database and script.) Every entry in the file will be in mytable
Is there a way to do this in psql from the command line? My intention is to run this as part of a bash script on our servers.
Related
I have an anonymous function containing a query within a FOR loop that executes 100 times, and I need to save the 100 result sets as 100 files on the remote client (not on the server).
It seems like the psql \copy meta-command should be the way to do this, but I'm at a loss. Something of this form, maybe?
\copy (anonymous_function_w/_FOR_loop_here) to 'filename.txt'
where filename.txt is built from the FOR loop variable's value in each iteration. That's important - the files on the remote client need to be named based on the FOR loop's variable.
Is there any way to pull this off? I suppose an alternative approach would be to UNION all 100 query results into one big result, with the FOR loop's variable value in one field, and then use bash scripting to split it into 100 appropriately named files. But my bash skills are pretty lame. If psql can do the job directly that would be great.
EDIT: I should add that here's what the FOR loop variable looks like:
FOR rec IN SELECT DISTINCT county FROM voter.counties
so the file name would be built from rec.county + '.txt'
The typical approach to this is to use a SQL statement that generates the necessary statements, spool the output into a script file, then run that file.
Something like:
-- prepare for a "plain" output without headers or something similar
\a
\t
-- spool the output into export.sql
\o export.sql
select format('\copy (select * from some_table where county = %L) to ''%s.txt''', county, county)
from (select distinct county from voter.counties) t;
-- turn spooling off
\o
-- run the generated file
\i export.sql
So for each county name in voters.counties the export.sql will contain:
\copy (select * from some_table where county = 'foobar') to 'foobar.txt'
I am using PostgreSQL 9.0.3. I have an Excel spreadsheet with lots of data to load into couple of tables in Windows OS.
I have written the script to get the data from input file and Insert into some 15 tables. This can't be done with COPY or Import. I named the input file as DATALD.
I find out the psql command -d to point the db and -f for the script sql. But I need to know the commands how to feed the input file along with the script so that the data gets inserted into the tables..
For example this is what I have done:
begin
for emp in (select distinct w_name from DATALD where w_name <> 'w_name')
--insert in a loop
INSERT INTO tblemployer( id_employer, employer_name,date_created, created_by)
VALUES (employer_id,emp.w_name,now(),'SYSTEM1');
Can someone please help?
For an SQL script you must ..
either have the data inlined in your script (in the same file).
or you need to utilize COPY to import the data into Postgres.
I suppose you use a temporary staging table, since the format doesn't seem to fit the target tables. Code example:
How to bulk insert only new rows in PostreSQL
There are other options like pg_read_file(). But:
Use of these functions is restricted to superusers.
Intended for special purposes.
In other statistical programs, it's possible to create a log file that shows the output issued as a result of a command. Is it possible to do something similar in SQL?
In particular, I'd like to have a single .sql file with many queries and to then output each result to a text file.
I'm using PostgreSQL and Navicat.
plpgsql function and COPY
One way would be to put the SQL script into a plpgsql function, where you can write the individual return values to files with COPY and compile a report from intermediary results just like you need it.
This has additional effect that may or may not be desirable. Like, you can grant or revoke permission to the whole function to arbitrary roles. Read about SECURITY DEFINER in the manual. And the syntax will be verified when you save the function - however, only superficially (there are plans to change that in the future). More details in this answer on dba.SE.
Basic example:
CREATE OR REPLACE FUNCTION func()
RETURNS void AS
$BODY$
BEGIN
COPY (SELECT * FROM tbl WHERE foo) TO '/path/to/my/file/tbl.csv';
COPY (SELECT * FROM tbl2 WHERE NOT bar) TO '/path/to/my/file/tbl2.csv';
END;
$BODY$
LANGUAGE plpgsql;
Of course, you need to have the necessary privileges in the database and in the file system.
Call it from the shell:
psql mydb -c 'SELECT func();'
psql switching between meta commands and SQL
#!/bin/sh
BASEDIR='/var/lib/postgresql/outfiles/'
echo "
\\o $OUTDIR/file1.txt \\\\\\\\ SELECT * FROM tbl1;
\\o $OUTDIR/file2.txt \\\\\\\\ SELECT * FROM tbl2;
\\o $OUTDIR/file3.txt \\\\\\\\ SELECT * FROM tbl3;" | psql event -p 5432 -t -A
That's right, 8 backslashes. Results from a double backslash that gets interpreted two times, so you have to double them two times.
I quote the manual about the meta-commands \o:
Saves future query results to the file filename or ...
and \\:
command must be either a command string that is completely parsable by
the server (i.e., it contains no psql-specific features), or a single
backslash command. Thus you cannot mix SQL and psql meta-commands with
this option. To achieve that, you could pipe the string into psql,
like this: echo '\x \\ SELECT * FROM foo;' | psql. (\\ is the
separator meta-command.)
Don't know about navicat, but you can do it with psql. Check the various --echo-X command-line options and the \o command if you just want temporary output to a file.
Is there any way, how to check, whether a variable has already been set in my environment?
Example:
\set table_name countries
\i queries.sql
queries.sql:
SELECT * FROM :table_name;
I want to make queries.sql to be called independently and use some default table name I would specify.
Is this possible or do I really need to create another SQL file through which I will call the queries (\i)?
My use case is usage of my SQL queries both in pgTAP unit tests (with some sample table names) and independently.
You could check the current value with:
SELECT :'table_name';
You can set it on the call to psql with something like --set='table_name' on the psql command line.
For a particular installation of my application, I need to create the database and the schema on the SQL server from the installer itself. I have a custom installer through which I have been able to detect and install the pre-requisites and the software. The user is prompted to give the IP of the database server and the username and password. Behind the scene, I create a connection and a command object. I keep the queries in different files. I use a reader and read the content of the file and set the content of the file to the CommandText of the command object. The typical content of the file is like following:
create database mydatabase
Go
Use mydatabase
Go
EXEC sp_MSforeachtable #command1 = "DROP TABLE ?"
Now the issue is the first statements get executed but it gives error after that. The error that is shown is: "syntax error near 'GO'". I tried removing the GO statement and also tried ending the sql statements with semi-colon. The error in this case is "Database 'mydatabase'does not exist. Make sure that the name is entered correctly.".
However if I keep a single statement in the file, it works fine.
Can somebody help me?
As you can see at http://technet.microsoft.com/en-us/library/aa258908%28SQL.80%29.aspx
Remarks
GO is not a Transact-SQL statement; it
is a command recognized by the osql
and isql utilities and SQL Query
Analyzer.
So this is the cause of your problems when you run it using the SqlCommand from .Net.
In my opinion you have two options:
1) Execute the instructions one by one. Maybe use a separator in your files, then split the SQL statements and execute them sequentially using a for/foreach.
2) Use Server class from SQL Server Management Objects (SMO) that should allow you to execute the script containing "Go" statements.
You can execute more than one sql command statement by simply adding a ";" at the end of each command instead of a "GO" statement.
Example:
cmd.CommandText = #" Update TableA Set ColumnA = 'Test' Where ID = 1;
Update TableB Set ColumnA = 'Second line' Where ID = 2;
";