I want to dump a select query to a tab-delimited text file using psql -F. However, this doesn't work:
psql -Umyuser mydb -F '\t' --no-align -c "select * from mytable" -o /tmp/dumpfile.txt
That makes the delimiter a literal \t. How do I get it to use real tabs instead?
I think you just need to use a literal tab. How this works depends on your shell. Have you seen this post?
In the bash shell you can do this with $'\t'.
Using the example in your question:
psql -Umyuser mydb -AF $'\t' --no-align -c "select * from mytable" -o /tmp/dumpfile.txt
From man bash:
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. [...] The expanded result is single-quoted, as if the dollar sign had not been present.
In Unix, you can also type
ctrl-V tab
ctrl-V tells the terminal not to interpret the next key.
This also works with carriage returns (^M) and many other special keys like arrow keys
In case if somebody looked for how to do it in the interactive shell:
\f '\t'
Related
This question already has answers here:
How to escape single quotes within single quoted strings
(25 answers)
Closed 1 year ago.
I'm trying to insert data into some tables of a Postgres database with a bash script. I got it working like this
psql -c '\copy raw.ap1_1 from file.csv with csv header;'
The problem is I wan't to give the copy command the Quote option, but I can't get it to work because of the quotes outside. I've tried wrapping single quotes in double quotes, double quotes in single quotes, using a backslash to escape quotes, etc.
What is want is something like this:
psql -c '\copy raw.ap1_1 from file.csv with csv header quote as '"';'
But of course I can't do that because the quotes are taken as part of the bash script. I either get a postgres syntax error, or I get a parsing EOF error (because the quotes override themselves).
This should work:
psql -c "\\copy raw.ap1_1 from file.csv (format 'csv', header, quote '\"')"
PostgreSQL's SQL offers an alternative quoting method called dollar quoting. That is probably the easiest way to do it.
psql -c '\copy raw.ap1_1 from file.csv with csv header quote as $$"$$'
Note that neither dollar signs nor double quotes are special inside shell single quotes. Also note that \copy doesn't end with a ;. Further note that specifying a quote character which is the same as the default seems a bit silly.
If you don't want to use dollar quoting, you could escape correctly at the shell level. You can't do that from within the shell's single quote so first you have to close the quotes, then print '"' each one escaped with a backslash. (If this were not the end of the line, you would then re-open the shell's single quote to finish the line.)
psql -c '\copy raw.ap1_1 from file.csv with csv header quote as '\'\"\'
First of all ,you will login . Then
Copy table_name from 'C:\\\\expdata_202111122321.csv' with delimiter ';' csv header encoding 'windows-1254';
I'm trying to export some data from a Firebird database, with FBExport to a CSV file.
The problem is I have two different errors. I spent a few hours to try different combinations:
Unknown switch -
Switches must begin with -
The command I tried:
fbexport -Sc -Q -F h:\AABBCC\export.csv -B -D h:\AABBCC\XXYYZZ.FDB -U "MMNNOO" -P "PPQQRR" -X "select PATIENTS.IPP, PATIENTS.NOM, PATIENTS.NOM_MARITAL, PATIENTS.NOM_USUEL, PATIENTS.PRENOMS, dmc.NUM_DOSSIER_PAPIER, dmc.NUM_DOSSIER_PAPIER_2, dmc.NUM_DOSSIER_PAPIER_3 from PATIENTS join dmc on PATIENTS.ipp = dmc.CODE where dmc.NUM_DOSSIER_PAPIER is not null or dmc.NUM_DOSSIER_PAPIER_2 is not null or dmc.NUM_DOSSIER_PAPIER_3 is not null;"
I absolutely don't understand what FBExport needs.
How can I export the data?
For me, the following command line works:
fbexport -Sc -D employee -U sysdba -P masterkey -F C:\Temp\export.csv -Q "select * from employee"
The problem in your original command was that you had a bare -Q, which caused the following -F to be interpreted as the argument of -Q, which then lead to h:\AABBCC\export.csv to be interpreted as an option, which then produced an error because it doesn't start with a -.
In addition, your command also had the following problems:
-B defines an alternative separator character for the produced CSV. It expects a separator character or TAB or \t for a tab. So, in similar vein as the previous problem, this would cause -D to be interpreted as an argument of -B, which then leads to h:\AABBCC\XXYYZZ.FDB to be interpreted as an option (without -).
-X is a primary option (like -S), to execute the query specified by -Q, instead of exporting (saving, -S). It doesn't accept a query text as argument, so the query text is also interpreted as an option (without -). This occurrence of -X should have been -Q.
I am having some difficulty with a Perl script that is invoked from cron.
One of the arguments for the script is a GPG passphrase. This is later interpolated into a string that is sent to the shell.
This particular passphrase contains an open parentheses, and the script fails with a "syntax error near unexpected token `(" error.
This is the offending part of the phrase:
m3(ÃÝ4úŤ;:q!
I have tried single and double quoting it before the value is used in the script, but that does not have an effect.
The phrase works correctly when invoking GPG directly from the shell, just not when it gets interpolated into the following:
`gpg --passphrase $gpgpp --batch -o $gpgofile -d $file`;
Where $gpgpp is the passphrase variable.
What is the correct way to escape this, and other potentially problematic characters?
The \Q and \E escape sequences are used to escape all "special" characters between them.
`gpg --passphrase \Q$gpgpp\E --batch -o $gpgofile -d $file`;
This should be done any time you have a variable that may contain characters which need to be escaped.
http://perldoc.perl.org/functions/quotemeta.html
Enclose the variables in quotes:
gpg --passphrase "$gpgpp" --batch -o "$gpgofile" -d "$file"
Otherwise, the shell will try to interpret contents of the variables as expressions.
I'm recording terminal sessions using the script command. Unfortunately the typescript output file contains many control-characters - for example from pressing the full screen command (F11) when in the vim editor or try it below.
script -f -t 2>${LOGNAME}-$(/bin/date +%Y%m%d-%H%M%S).time -a ${LOGNAME}-$(/bin/date +%Y%m%d-%H%M%S).session
vi abc.log
#write something and save
#:x to quit vi
ctrl + d to quit script
The script output hostname-datetime.session contais too many vi control-characters.
I found a perl script in commandlinefu, which can remove these control characters from the typescript.
I am actually doing this replacement in C, and the program runs on a chroot envrioment, where the perl is not avaliable.
Question: Is there a a way to translate the following perl command to sed ?
cat typescript | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > typescript-processed
if you ONLY want printable ascii :
LC_ALL=C tr -cd ' -~\n\t' < typescript > typescript_printable_ascii_only
why this works? all printable ("normal") ascii are between Space and Tilde
And in addition you need Newline and Tab.
So ' -~\n\t' covers all printable "normal" ascii characters. And tr -d 'chars' deletes all chars, -c takes the opposite of the range given (so everything except 'chars').
=> This LC_ALL=C tr -cd ' -~\n\t' deletes everything except the normal ascii characters (including newline and tab) (I force the locale to be 'C' to be sure we are in the right locale when calling "tr")
This works well for me with GNU sed (or gsed on a Mac):
sed -re 's/\x1b[^m]*m//g' typescript | col -b
I created a sample typescript, and since I'm using a relatively advanced shell prompt, it's full of control characters, and the perl script in the OP doesn't actually work, so rather than converting I had to come up with my own.
Looking at the typescript with hexdump -C, it seems that all control sequences start with \x1b (the Escape character, or ^[), and end with the letter "m". So in sed I use a simple replacement from ^[ until m, normally written as \x1b.*?m but since sed doesn't support the ? symbol to make a pattern non-greedy, I used [^m]*m to emulate non-greedy matching.
I have a somewhat detailed query in a script that uses ? placeholders. I wanted to test this same query directly from the psql command line (outside the script). I want to avoid going in and replacing all the ? with actual values, instead I'd like to pass the arguments after the query.
Example:
SELECT *
FROM foobar
WHERE foo = ?
AND bar = ?
OR baz = ? ;
Looking for something like:
%> {select * from foobar where foo=? and bar=? or baz=? , 'foo','bar','baz' };
You can use the -v option e.g:
$ psql -v v1=12 -v v2="'Hello World'" -v v3="'2010-11-12'"
and then refer to the variables in SQL as :v1, :v2 etc:
select * from table_1 where id = :v1;
Please pay attention to how we pass string/date values using two quotes " '...' " But this way of interpolation is prone to SQL injections, because it's you who's responsible for quoting. E.g. need to include a single quote? -v v2="'don''t do this'".
A better/safer way is to let PostgreSQL handle it:
$ psql -c 'create table t (a int, b varchar, c date)'
$ echo "insert into t (a, b, c) values (:'v1', :'v2', :'v3')" \
| psql -v v1=1 -v v2="don't do this" -v v3=2022-01-01
Found out in PostgreSQL, you can PREPARE statements just like you can in a scripting language. Unfortunately, you still can't use ?, but you can use $n notation.
Using the above example:
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3 ;
EXECUTE foo('foo','bar','baz');
DEALLOCATE foo;
In psql there is a mechanism via the
\set name val
command, which is supposed to be tied to the -v name=val command-line option. Quoting is painful, In most cases it is easier to put the whole query meat inside a shell here-document.
Edit
oops, I should have said -v instead of -P (which is for formatting options) previous reply got it right.
You can also pass-in the parameters at the psql command-line, or from a batch file. The first statements gather necessary details for connecting to your database.
The final prompt asks for the constraint values, which will be used in the WHERE column IN() clause. Remember to single-quote if strings, and separate by comma:
#echo off
echo "Test for Passing Params to PGSQL"
SET server=localhost
SET /P server="Server [%server%]: "
SET database=amedatamodel
SET /P database="Database [%database%]: "
SET port=5432
SET /P port="Port [%port%]: "
SET username=postgres
SET /P username="Username [%username%]: "
SET /P bunos="Enter multiple constraint values for IN clause [%constraints%]: "
ECHO you typed %constraints%
PAUSE
REM pause
"C:\Program Files\PostgreSQL\9.0\bin\psql.exe" -h %server% -U %username% -d %database% -p %port% -e -v v1=%constraints% -f test.sql
Now in your SQL code file, add the v1 token within your WHERE clause, or anywhere else in the SQL. Note that the tokens can also be used in an open SQL statement, not just in a file. Save this as test.sql:
SELECT * FROM myTable
WHERE NOT someColumn IN (:v1);
In Windows, save the whole file as a DOS BATch file (.bat), save the test.sql in the same directory, and launch the batch file.
Thanks for Dave Page, of EnterpriseDB, for the original prompted script.
I would like to offer another answer inspired by #malcook's comment (using bash).
This option may work for you if you need to use shell variables within your query when using the -c flag. Specifically, I wanted to get the count of a table, whose name was a shell variable (which you can't pass directly when using -c).
Assume you have your shell variable
$TABLE_NAME='users'
Then you can get the results of that by using
psql -q -A -t -d databasename -c <<< echo "select count(*) from $TABLE_NAME;"
(the -q -A -t is just to print out the resulting number without additional formatting)
I will note that the echo in the here-string (the <<< operator) may not be necessary, I originally thought the quotes by themselves would be fine, maybe someone can clarify the reason for this.
It would appear that what you ask can't be done directly from the command line. You'll either have to use a user-defined function in plpgsql or call the query from a scripting language (and the latter approach makes it a bit easier to avoid SQL injection).
I've ended up using a better version of #vol7ron answer:
DO $$
BEGIN
IF NOT EXISTS(SELECT 1 FROM pg_prepared_statements WHERE name = 'foo') THEN
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3;
END IF;
END$$;
EXECUTE foo('foo','bar','baz');
This way you can always execute it in this order (the query prepared only if it does not prepared yet), repeat the execution and get the result from the last query.