I'm attempting to spy on non-SSL PostgreSQL traffic using tshark using the following command:
# tshark -f 'tcp dst port 5432' -O PGSQL \
-d 'tcp.port==5432,pgsql' -T fields -e pgsql.query
I am able to see SQL queries, but all the actual values/parameters are missing (instead replaced with placeholders $1, $2, $3 etc). Example output is as follows:
...
INSERT INTO mdl_logstore_standard_log
(eventname,component,action,target,objecttable,objectid,crud,edulevel,
contextid,contextlevel,contextinstanceid,userid,courseid,relateduserid,
anonymous,other,timecreated,origin,ip,realuserid)
VALUES
($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20),
($21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40)
INSERT INTO mdl_backup_files_temp (contextid,component,filearea,itemid,info,backupid)
VALUES($1,$2,$3,$4,$5,$6) RETURNING id RELEASE SAVEPOINT
moodle_pg_savepoint; SAVEPOINT moodle_pg_savepoint SELECT * FROM mdl_backup_ids_temp
WHERE backupid = $1 AND itemname = $2 AND itemid = $3 INSERT INTO
mdl_backup_files_temp (contextid,component,filearea,itemid,info,backupid)
VALUES($1,$2,$3,$4,$5,$6) RETURNING id RELEASE SAVEPOINT moodle_pg_savepoint;
SAVEPOINT moodle_pg_savepoint
...
What am I missing here - and how can I view the values/parameters as well ?
In order to see the values passed in to a prepared statement, you'll need to set log_statement to either mod or all. log_statement prints the statement that is about to be executed, and includes the parameters/arguments used.
I think the easiest way to turn it on is by doing:
psql -c "ALTER SYSTEM SET log_statement TO 'all'"
psql -c "SELECT pg_reload_conf()"
From there, you should be able to view the parameters.
Bear in mind, this has the potential to generate a lot of traffic, so you'll sent to set it back to the previous value once you're done (you can get the current value by calling psql -c "SHOW log_statement" before you do the two commands above).
In order to see the parameters, you would give tshark another field to print:
-e pgsql.query -e pgsql.val.data
But this is going to be mess, especially if you use prepared statements. You should really just figure out what you are doing wrong with log_statement='all', it will log all statements not just a sample of them. Maybe you have that setting countermanded per user, per database, or per connection.
I would like to have a .psqlrc file with default values, and be able to override these values from psql's command line.
For example :
have some values set in .psqlrc :
-- .psqlrc :
-- "user#database # " in bold green
\set PROMPT1 '%[%033[1;32;40m%]%n#%/%[%033[0m%]% > '
-- store command history in home directory, with a per database file :
\set HISTFILE ~/.psql-history- :DBNAME
in some wrapper script master-psql.sh, which connects as user postgres, be able to override these values :
# master-psql.sh :
# when using this script, change color to red, change history file location :
psql -U postgres \
-v PROMPT1='%[%033[1;31;40m%]%n#%/%[%033[0m%]% # ' \
-v HISTFILE='/some/other/place/.psql_history_postgres'
The above does not work, because the the -v ... argument is executed before the .psqlrc file is loaded, and the instruction in .psqlrc overwrites the existing value.
Question
Is there a way to instruct psql to run a set of commands after loading its .psqlrc file(s),
or to have .psqlrc execute some \set or \pset command only if value is not set ?
You could write the instructions not to overwrite those variables if already set into the .psqlrc file itself:
\if :{?HISTFILE}
\else
\set HISTFILE ~/.psql-history- :DBNAME
\endif
If you can't get your system psqlrc to cooperate with you, then might need to copy and modify it and then bypass the original. You need at least v11 for the :{? construct to work.
The problem is that PROMPT1 has a compiled-in default even in the absence of RC file processing, so you might need to test that against the compiled-in string, rather than test for being defined. So I think that would end up with something like this:
select :'PROMPT1'='%/%R%x%# ' as default_prompt \gset
\if :default_prompt
\set PROMPT1 '%[%033[1;32;40m%]%n#%/%[%033[0m%]% > '
\endif
Note that the compiled in default changed in v13, so if you want to work with older versions as well, you would need to do something more complicated.
From here:
https://www.postgresql.org/docs/current/app-psql.html
Environment
[...]
PSQLRC
Alternative location of the user's .psqlrc file. Tilde (~) expansion is performed.
So create an alternate .psqlrc file and set thePSQLRC environment variable for the script to override your default.
I'm currently trying to use exiftool on Windows command prompt to read meta data from multiple files, then output to a single text file.
The exact command I last tried looked like this:
exiftool.exe -FileName -GPSPosition -CreateDate -d "%m:%d:%Y %H:%M:%S" -c "%d° %d' %.2f"\" -charset UTF-8 -ext jpg -w _Coordinate_Date.txt S:\Nick\Test\
When I run this, I get 7 individual text files with the content for one corresponding file in each of them. However, I simply want to output all of it to one single text file. Any help is greatly appreciated
The -w (textout) option can only be used to write multiple files. It is not meant to be used to output to a single file. As per the docs on -w:
It is not possible to specify a simple filename as an argument -- creating a single output file from multiple source files is typically done by shell redirection
Which is what you're doing with the >> ./output.txt part of your command. The -w _Coordinate_Date.txt isn't doing anything and I would think throw an Invalid TAG name: "w _Coordinate_Date.txt" error if quoted together like that as it gets treated as a single arugment. The -w option requires two arguments, the -w and either an extension or a format string.
I actually figured it out, if you wrap the entire -w _Coordinate_Date.txt command in quotations and append it to a file, you can throw all of the output into one text file.
i.e. "-w _Coordinate_Date.txt >> ./output.txt"
I have a directory with at least 6 function files.
I'm using psql and I need to be able to source (initialize ?) all function files at once.
I'm sure making a single function and call all others like SELECT fn1, fn2 isn't going to work.
Doing \i functions_folder/fn1.sql 6 times isn't ideal either.
So is there a way I can (maybe) \i functions/*.sql? Doing that currently gives me
functions/*.sql: No such file or directory
I'm using psql on postgresql 9.6.2
If you are using *nix OS:
postgres=# \! cat ./functions/*.sql > all_functions.sql
postgres=# \i all_functions.sql
For Windows try to find the analogue of the cat (as I remember it is copy command with some flags)
PS I have the feeling that it could be done by using backquotes:
Within an argument, text that is enclosed in backquotes (`) is taken as a command line that is passed to the shell. The output of the command (with any trailing newline removed) replaces the backquoted text.
Documentation
But still have no idea how. Probably somebody more experienced will provide a hint...
Create a wrapper that contains the \i commands, and \i the wrapper.
I am not a programmer, I am struggling a bit with this.
I have a batch file connecting to my PostgreSQL server, and then open a sql script. Everything works as expected. My question is how to pass a variable (if possible) from one to the other.
Here is my batch file:
set PGPASSWORD=xxxx
cls
#echo off
C:\Progra~1\PostgreSQL\8.3\bin\psql -d Total -h localhost -p 5432 -U postgres -f C:\TotalProteinImport.sql
And here's the script:
copy totalprotein from 'c:/TP.csv' DELIMITERS ',' CSV HEADER;
update anagrafica
set pt=(select totalprotein.resultvalue from totalprotein where totalprotein.accessionnbr=anagrafica.id)
where data_analisi = '12/23/2011';
delete from totalprotein;
This is working great, now the question is how could I pass a variable that would carry the date for data_analisi?
Like in the batch file, "Please enter date", and then the value is passed to the sql script.
You could create a function out of your your SQL script like this:
CREATE OR REPLACE FUNCTION f_myfunc(date)
RETURNS void AS
$BODY$
CREATE TEMP TABLE t_tmp ON COMMIT DROP AS
SELECT * FROM totalprotein LIMIT 0; -- copy table-structure from table
COPY t_tmp FROM 'c:/TP.csv' DELIMITERS ',' CSV HEADER;
UPDATE anagrafica a
SET pt = t.resultvalue
FROM t_tmp t
WHERE a.data_analisi = $1
AND t.accessionnbr = a.id;
-- Temp table is dropped automatically at end of session
-- In this case (ON COMMIT DROP) after the transaction
$BODY$
LANGUAGE sql;
You can use language SQL for this kind of simple SQL batch.
As you can see I have made a couple of modifications to your script that should make it faster, cleaner and safer.
Major points
For reading data into an empty table temporarily, use a temporary table. Saves a lot of disc writes and is much faster.
To simplify the process I use your existing table totalprotein as template for the creation of the (empty) temp table.
If you want to delete all rows of a table use TRUNCATE instead of DELETE FROM. Much faster. In this particular case, you need neither. The temporary table is dropped automatically. See comments in function.
The way you updated anagrafica.pt you would set the column to NULL, if anything goes wrong in the process (date not found, wrong date, id not found ...). The way I rewrote the UPDATE, it only happens if matching data are found. I assume that is what you actually want.
Then ask for user input in your shell script and call the function with the date as parameter. That's how it could work in a Linux shell (as user postgres, with password-less access (using IDENT method in pg_haba.conf):
#! /bin/sh
# Ask for date. 'YYYY-MM-DD' = ISO date-format, valid with any postgres locale.
echo -n "Enter date in the form YYYY-MM-DD and press [ENTER]: "
read date
# check validity of $date ...
psql db -p5432 -c "SELECT f_myfunc('$date')"
-c makes psql execute a singe SQL command and then exits. I wrote a lot more on psql and its command line options yesterday in a somewhat related answer.
The creation of the according Windows batch file remains as exercise for you.
Call under Windows
The error message tells you:
Function tpimport(unknown) does not exist
Note the lower case letters: tpimport. I suspect you used mixe case letters to create the function. So now you have to enclose the function name in double quotes every time you use it.
Try this one (edited quotes!):
C:\Progra~1\PostgreSQL\8.3\bin\psql -d Total -h localhost -p 5432 -U postgres
-c "SELECT ""TPImport""('%dateimport%')"
Note how I use singe and double quotes here. I guess this could work under windows. See here.
You made it hard for yourself when you chose to use mixed case identifiers in PostgreSQL - a folly which I never tire of warning against. Now you have to double quote the function name "TPImport" every time you use it. While perfectly legit, I would never do that. I use lower case letters for identifiers. Always. This way I never mix up lower / upper case and I never have to use double quotes.
The ultimate fix would be to recreate the function with a lower case name (just leave away the double quotes and it will be folded to lower case automatically). Then the function name will just work without any quoting.
Read the basics about identifiers here.
Also, consider upgrading to a more recent version of PostgreSQL 8.3 is a bit rusty by now.
psql supports textual replacement variables. Within psql they can be set using \set and used using :varname.
\set xyz 'abcdef'
select :'xyz';
?column?
----------
abcdef
These variables can be set using command line arguments also:
psql -v xyz=value
The only problem is that these textual replacements always need some fiddling with quoting as shown by the first \set and select.
After creating the function in Postgres, you must create a .bat file in the bin directory of your Postgres version, for example C:\Program Files\PostgreSQL\9.3\bin. Here you write:
#echo off
cd C:\Program Files\PostgreSQL\9.3\bin
psql -p 5432 -h localhost -d myDataBase -U postgres -c "select * from myFunction()"