We were trying to use the \COPY command in Knex.raw. We connect via heroku/postgres and the \COPY is the only way to bulk import data as we don't have superuser rights. The backslash before the copy is being removed or multiplied depending on what you try.
So far we've tried to escape it using
1. \b
2. \
3. String.fromCharCode(92)
4. concatenation
The full function we're using is...
knex.raw("\COPY tablename (column1, column2, ...columnN) FROM STDIN " + csvValues).then(function(response){
console.log('the response from the copy', response);
});
We've tested this exact function within our pg:psql environment and it works perfectly. If you know of a better way of doing this, please let us know!
Regards,
Brandon
\copy is a psql (client) command, which wraps Postgres's COPY (server). It sounds like perhaps you're trying to combine the two, and that's where the problem is.
Since knex.js is a query builder, it wouldn't know about \copy -- \COPY would not be valid SQL to use in a client scenario.
You mention that you don't have superuser access, which, if you want to reference a file, is required for COPY.
If you're unable to use COPY for that reason, you would instead need to use psql and leverage its \copy command, for which knex.js would likely not be the tool to use. Rather, you would need to access it via the invocation of a shell command.
Perhaps this batch import of data could be done outside of Javascript entirely? i.e. via a bash script that used psql directly?
Related
I have two DB instances both running PG 9.4
When i issue the COPY command in one it will return with the number of rows affected, however in the second DB which is set up the same it will not.
I see nothing in the config that is different or may affect such. The imports do not error and import successfully on both accounts.
The Documentation states it should return as long as its not stdout.
This line in the documentation looks pertinent, but i'm not sure it applies to my situation.
Do not confuse COPY with the psql instruction \copy. \copy invokes COPY FROM STDIN or COPY TO STDOUT, and then fetches/stores the data in a file accessible to the psql client. Thus, file accessibility and access rights depend on the client rather than the server when \copy is used.
The command i'm issuing is:
COPY [tablename] from '/var/lib/pgsql/datafile.csv'
At the moment i'm down to looking at putty session variables, but i'm not sure this is the way to go.
Does anyone have any ideas as to why this may be happening?
When psql is quiet, it doesn't display these messages.
The quiet mode is activated with -q or \set QUIET on
Example:
test=# copy test to '/tmp/foo';
COPY 8
test=# \set QUIET on
test=# copy test to '/tmp/foo';
test=#
I have to copy few postgres table contents into a CSV file for further processing using Talend.
When I tried
copy table_name to ‘/tmp/export/table_name.csv’ with CSV;
with 'tpostgresqlRow' component, it threw error saying "Need superuser access", which I wouldn't be getting.
As an alternative, I tried "\copy" command as
\\copy table_name to ‘/tmp/export/table_name.csv’ with CSV;"
(first slash is used to escape the other). Still it threw error saying "Syntax error at '\'".
I tried with tpostgresqlBulkExec component as well, which internally uses "copy" command, and it as well threw error saying "Need superuser access".
Is there a way out to execute this postgres "copy" command in bulk using Talend?
Any help would be much appreciated.
The copy command needs access to both the postgres table and the local file system.
I think your problem is caused because you do not have access to the local folder. you need to sort that out, or try a different folder that you know you have access to.
with the same user, if you try linux command
touch /tmp/export/test.csv
you may see an error.
the \copy command will fail because to use that you first need to be using a tool called psql.
I got confirmation from Talend Support Team that tPostgresqlRow component would not support 'COPY' command. I have resolved my case by installing PSQL client and calling the same using tSSH component in Talend to execute the desired 'COPY' commands.
I am making an automated script from terminal that creates a file with the output of \l
But I do not know where the \o command in postgresql prints out the file that it has made. The documentation doesn't inform where.
I did read this, but no luck:
Sincerely
\o points at the named file in current working directory of psql. As you found out, this has some issues in automated environments.
This means you have basically two options:
use absolute paths with \o
Alternatively you can use \cd to set your current working directory inside your psql script.
In your particular case, however, you know that psql -l gives you the same info? That may be easier to manage in an automated environment.
I am writing a KSH script to accept a .sql file parameter followed by each schema that file is to be compiled in. The script assumes the .sql file is a DB2 9.7 procedure.
I believe I am stuck on the syntax of the db2 command for the termination character. We always use the at symbol ('#'), however the following snippet fails with error "DB21001E The option "-#" specified after the 'db2' command or in the DB2OPTIONS variable is incorrect." Any assistance would be greatly appreciated.
#!/bin/ksh
. $IBM_DB_DIR/db2profile
db2 connect to dwdev3
const_compil_string_suffix="-vtd# -f ../../stored_procedures/"
script_name="ETL.THING.sql"
db2 "$const_compil_string_suffix$script_name"
db2 terminate
I have confirmed that the resulting string command above the 'terminate' does work at linux prompt compiling the procedure as expected:
db2 -vtd# -f ../../stored_procedures/ETL.THING.sql
Thank you in advance.
Try losing the quotation marks:
db2 ${const_compil_string_suffix}${script_name}
I suspect the option string -vtd# might be the culprit.
When using single character options most unix command accept two types of options, those with and those without arguments.
Apparently the # character is being processed as an option argument, however the error message indicates it is seen as an option on its own.
Try separating the options, as in: -v -t -d#, reorder them. or remove one of the -t or -v options.
Beware though. I have no knowledge about the db2 cli. Experiment at your own risk.
Was the -t option added later?
I want to export the table into a .CSV file using \COPY command. I can able to do that as a stand-alone command. I can't embed the same line inside a function in PostgreSQL. Actually that call should come from a ECPG. I chose \COPY over COPY command as I don't have super-user account! Please guide me on this.
Thanks and Regards,
Siva.
Unfortunately, all of the \ commands are psql commands. You can run psql -E to see what those commands expand into and get sent to the server as, but if you need a super-user account to run COPY, you're going to need a super-user account to do this.
If you are talking about using copy with plpgsql, I think this might help:
Dynamically-generated table-name in PostgreSQL COPY command