How can I save my query results in a CSV file via the Impala Shell.
My Code:
impala-shell -q "use test;
select * from teams;
-- From this point I need to save the query results to /Desktop (for example).
"
The problem that I am getting is that I have to select the DB first and then operate the query, but I only see syntax commands that directly uses the query instead of using the DB and then the query.
Found it.
impala-shell -B -o output.csv --output_delimiter=',' -q "use test;
select * from teams;"
You can use
impala-shell -B -q "select * from anylbi.tablename_1;" -o extract.csv
--print_header '--output_delimiter=,'
if you have a particular server you're trying to connect you can use the -i option and then the server name after impala-shell
eg:
Server : SERVER_NAME
Output File : output.csv
Database : test
Table name : teams
Delimiter : |
impala-shell -k -i SERVER_NAME --ssl -B -o output.csv --print_header --output_delimiter=| -q "use test; select * from teams";
Related
I have following script that I am using to copy select query result in csv file.
psql -h hostname -U username -d dbname -t -A -F"," --c "some query with multiple joins and conditions" > "E:\output.csv"
while executing the script it is creating the file with name output.csv but there is no data in file.
I have tried with sample query simple join operation with 3 tables for limited data and it is working fine.
But with actual query it keeps executing for the couple of hours and ended with the empty file. Which is dissatisfying to get empty file after waiting such a long period.
any suggestions or corrections?
In postgresql 9.3.1, when interactively developing a query using the psql command, the end result is sometimes to write the query results to a file:
boron.production=> \o /tmp/output
boron.production=> select 1;
boron.production=> \o
boron.production=> \q
$ cat /tmp/output
?column?
----------
1
(1 row)
This works fine. But how can I get the query itself to be written to the file along with the query results?
I've tried giving psql the --echo-queries switch:
-e, --echo-queries
Copy all SQL commands sent to the server to standard output as well.
This is equivalent to setting the variable ECHO to queries.
But this always echoes to stdout, not to the file I gave with the \o command.
I've tried the --echo-all switch as well, but it does not appear to echo interactive input.
Using command editing, I can repeat the query with \qecho in front of it. That works, but is tedious.
Is there any way to direct an interactive psql session to write both the query and the query output to a file?
You can try redirecting the stdout to a file directly from your shell (Win or Linux should work)
psql -U postgres -c "select 1 as result" -e nomedb >> hello.txt
This has the drawback of not letting you see the output interactively. If that's a problem, you can either tail the output file in a separate terminal, or, if in *nix, use the tee utility:
psql -U postgres -c "select 1 as result" -e nomedb | tee hello.txt
Hope this helps!
Luca
I know this is an old question, but at least in 9.3 and current versions this is possible using Query Buffer meta-commands shown in the documentation or \? from the psql console: https://www.postgresql.org/docs/9.3/static/app-psql.html
\w or \write filename
\w or \write |command
Outputs the current query buffer to the file filename or pipes it to the shell command command.
Please try this format as I got the output from the same:
psql -h $host -p $port -q -U $user -d $Dbname -c "SELECT \"Employee-id\",\"Employee-name\" FROM Employee_table" >> Employee_Date.csv
I need the output in a CSV file.
I want to store following postgreSQL query result in a variable. I am writing command on the shell script.
psql -p $port -c "select pg_relation_size ('tableName')" postgres
I need variable to save the result on a file. I have tried following but it is not working
var= 'psql -p $port -c "select pg_relation_size ('tableName')" '
Use a shell HERE document like:
#!/bin/sh
COUNT=`psql -A -t -q -U username mydb << THE_END
SELECT count (DISTINCT topic_id) AS the_count
FROM react
THE_END`
echo COUNT=${COUNT}
The whole psql <<the_end ... stuff here ... the_end statement is packed into backticks
the output of the execution of the statement inside the backticks is used as a value for the COUNT shell variable
The -A -t -q are needed to suppress column headers and error output
inside a here document, shell variable substitution works, even in single quotes!
So, you could even do:
#!/bin/sh
DB_NAME="my_db"
USR_NAME="my_name"
TBL_NAME="my_table"
COL_NAME="my_column"
COUNT=`psql -A -t -q -U ${USR_NAME} ${DB_NAME} << THE_END
SELECT COUNT(DISTINCT ${COL_NAME} ) AS the_count
FROM ${TBL_NAME}
THE_END`
echo COUNT=${COUNT}
to run a query inline you have to wrap it in grave accents, not single quotes:
$ vim `which fancyexecfileinpath`
psql lets you run queries from command line, but I guess you should be inputting complete information. you might be missing the database name.
postgres#slovenia:~$ psql -d mydbname -c "select * from applications_application;"
postgres#slovenia:~$ specialvar=`psql -d flango -c "select * from applications_application;"`
postgres#slovenia:~$ echo $specialvar
id | name | entities | folder | def_lang_id | ... | 2013-07-09 15:16:57.33656+02 | /img/app3.png (1 row)
postgres#slovenia:~$
notice the grave accents when assigning it to specialvar
otherwise you'll be setting it to a string.
There shouldn't be any space between the variable and the equals sign ("=") and the value ( http://genepath.med.harvard.edu/mw/Bash:HOW_TO:_Set_an_environment_variable_in_the_bash_shell )
For instance, I have a table stores value:
select * from myvalue;
val
-------
12345
(1 row)
How can I save this 12345 into a variable in postgresql or shell script?
Here's what I tried in my shell script:
var=$(psql -h host -U user -d db <<SQLSTMT
SELECT * FROM myvalue;
SQLSTMT)
but echo $var gives me:
val ------- 12345 (1 row)
I've also tried
\set var (select * from myvalue)
in psql and when I type \set it lists:
var = '(select*frommyvalue)'
No, no, no! Use "raw data" switch from psql, like "-t" or "\t" and pipe the query to psql instead of parsing ascii-table, come on :-)
echo 'select * from myvalue;' | psql -t -h host -U user -d db
If you really need parse psql output, you could also use -H switch ( turns on HTML output ), and parse it with some perl module for parsing html tables, I used that once or twice.. Also, you may want to use a pgpass file and ~/.psqlrc for some defaults, like default DB to connect, when not specified.
psql has a -c/--command= option to accept SQL from the command line, and -t/--tuples-only option to control output formatting.
$ psql -c 'select 1+1'
?column?
----------
2
(1 row)
$ psql -t -c 'select 1+1'
2
$ VALUE=`psql -t -c 'select 1+1'`
$ echo $VALUE
2
var=`psql -Atc "select 1;"`
echo $var
1
In this answer I explain one way to do it, using a co-process to communicate back-and-forth with psql. That's overkill if all you need is to run a query and get a single result, but might be good to know if you're shell scripting with psql.
You can filter the result you get with your psql command:
var=$(psql -h host -U user -d db <<SQLSTMT
SELECT * FROM myvalue;
SQLSTMT)
var=$(cut -d' ' -f3 <<<$var)
None of these worked for me, but this did:
median_avm=psql "host=${dps1000} port=#### dbname=### user=${reduser} password=${redpass}" -c "SELECT AVG(column) FROM db.table;" -t
using a source file with ${dps1000}, ${reduser}, ${redpass} defined and manually entering port and dbname
I'm using psql's \dt to list all tables in a database and I need to save the results.
What is the syntax to export the results of a psql command to a file?
From psql's help (\?):
\o [FILE] send all query results to file or |pipe
The sequence of commands will look like this:
[wist#scifres ~]$ psql db
Welcome to psql 8.3.6, the PostgreSQL interactive terminal
db=>\o out.txt
db=>\dt
Then any db operation output will be written to out.txt.
Enter '\o' to revert the output back to console.
db=>\o
The psql \o command was already described by jhwist.
An alternative approach is using the COPY TO command to write directly to a file on the server. This has the advantage that it's dumped in an easy-to-parse format of your choice -- rather than psql's tabulated format. It's also very easy to import to another table/database using COPY FROM.
NB! This requires superuser or pg_write_server_files privileges and will write to a file on the server.
Example: COPY (SELECT foo, bar FROM baz) TO '/tmp/query.csv' (format csv, delimiter ';')
Creates a CSV file with ';' as the field separator.
As always, see the documentation for details
Use o parameter of pgsql command.
-o, --output=FILENAME send query results to file (or |pipe)
psql -d DatabaseName -U UserName -c "SELECT * FROM TABLE" -o /root/Desktop/file.txt
\copy which is a postgres command can work for any user. Don't know if it works for \dt or not, but general syntax is reproduced from the following link Postgres SQL copy syntax
\copy (select * from tempTable limit 100) to 'filenameinquotes' with header delimiter as ','
The above will save the output of the select query in the filename provided as a csv file
EDIT:
For my psql server the following command works this is an older version v8.5
copy (select * from table1) to 'full_path_filename' csv header;
Use the below query to store the result in a CSV file
\copy (your query) to 'file path' csv header;
Example
\copy (select name,date_order from purchase_order) to '/home/ankit/Desktop/result.csv' cvs header;
Hope this helps you.
If you got the following error
ufgtoolspg=> COPY (SELECT foo, bar FROM baz) TO '/tmp/query.csv' (format csv, delimiter ';');
ERROR: must be superuser to COPY to or from a file
HINT: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
you can run it in this way:
psql somepsqllink_or_credentials -c "COPY (SELECT foo, bar FROM baz) TO STDOUT (format csv, delimiter ';')" > baz.csv
COPY tablename TO '/tmp/output.csv' DELIMITER ',' CSV HEADER;
this command is used to store the entire table as csv
I assume that there exist some internal psql command for this, but you could also run the script command from util-linux-ng package:
DESCRIPTION
Script makes a typescript of everything printed on your terminal.
This approach will work with any psql command from the simplest to the most complex without requiring any changes or adjustments to the original command.
NOTE: For Linux servers.
Save the contents of your command to a file
MODEL
read -r -d '' FILE_CONTENT << 'HEREDOC'
[COMMAND_CONTENT]
HEREDOC
echo -n "$FILE_CONTENT" > sqlcmd
EXAMPLE
read -r -d '' FILE_CONTENT << 'HEREDOC'
DO $f$
declare
curid INT := 0;
vdata BYTEA;
badid VARCHAR;
loc VARCHAR;
begin
FOR badid IN SELECT some_field FROM public.some_base LOOP
begin
select 'ctid - '||ctid||'pagenumber - '||(ctid::text::point) [0]::bigint
into loc
from public.some_base where some_field = badid;
SELECT file||' '
INTO vdata
FROM public.some_base where some_field = badid;
exception
when others then
raise notice 'Block/PageNumber - % ',loc;
raise notice 'Corrupted id - % ', badid;
--return;
end;
end loop;
end;
$f$;
HEREDOC
echo -n "$FILE_CONTENT" > sqlcmd
Run the command
MODEL
sudo -u postgres psql [some_db] -c "$(cat sqlcmd)" >>sqlop 2>&1
EXAMPLE
sudo -u postgres psql some_db -c "$(cat sqlcmd)" >>sqlop 2>&1
View/track your command output
cat sqlop
Done! Thanks! =D
Approach for docker
via psql command
docker exec -i %containerid% psql -U %user% -c '\dt' > tables.txt
or query from sql file
docker exec -i %containerid% psql -U %user% < file.sql > data.txt