How to copy CSV data to PostgreSQL using PowerShell - postgresql

I'm new to PowerShell. I'm able to establish connection to my PostgreSQL server and perform operation of deleting data from existing table. What I'm not able to perform is the copy statement. Below is my code.
$DBConnectionString = "Driver={PostgreSQL UNICODE(x64)};Server=mysvr;Port=5420;Database=mydb;Uid=myuser;Pwd=mypwd;Options='autocommit=off';"
$DBConn = New-Object System.Data.Odbc.OdbcConnection;
$DBConn.ConnectionString = $DBConnectionString;
$DBConn.Open();
$DeleteexistingdataDml3 = "delete from table1;delete from table2;"
$DBCmd = $DBConn.CreateCommand();
$DBCmd.CommandText = $DeleteexistingdataDml3;
$DBCmd.ExecuteReader();
I'm getting error while performing the copy statement.
$Copydatatotable = "\copy table1 FROM '\\filedolder\table1data.csv' DELIMITER AS '|' CSV NULL AS '';
\copy table2 FROM '\\filedolder\table2data.csv' DELIMITER AS '|' CSV NULL AS '';"
$DBCmd = $DBConn.CreateCommand();
$DBCmd.CommandText = $Copydatatotable;
$DBCmd.ExecuteReader();
Error:
Exception calling "ExecuteReader" with "0" argument(s): "ERROR [42601] ERROR: syntax error at or near "";
Thank you.

My understanding is that \COPY is a client command. I expect that means that it's understood by psql only. A generic ODBC provider won't understand it.
If you use the Npgsql .Net client, you may be able to use the PosgresSQL COPY command with it, but the \COPY command is for the PostgreSQL client psql only.
If you must use the ODBC connection, your best bet is likely to build an INSERT command with parameters and inserting each row of your CSV file one row at a time. In general, that would look similar to this answer inside a loop, but using Powershell instead of C#, of course.

Related

mysql connector python execute multiple statement

I would like to execute multiple statements by mysql connector. The code is as below,
import mysql.connector
conn = mysql.connector.connect(user='root', password='******',
database='dimensionless_ideal')
cursor = conn.cursor()
sql = ("Select * From conditions_ld; "
"Select * From conditions_fw; "
"Select * From results")
cursor.execute(sql, multi=True)
conn.close()
When I run this python script, it is keep running for a very long time unless I kill it. No output and error info. What is the problem?
You should take a look here : https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
It seems that when you use the Multi=True option, the cursor.execute returns an iterator. There is an example on how to use it.

TSQL - Bulk Insert

BULK
INSERT Table1
FROM 'C:\\Table1.txt' --location with filename
WITH
.
.
.
The Table1.txt is on a Windows Server. How should I write the WITH-Statement (Connection String)?
Try this:
BULK INSERT
Table1
FROM
'\\fullpath\Table1.txt'
WITH
(FIELDTERMINATOR = ',', ROWTERMINATOR = '\n')
Please Note: SQL Server NT service should have access to the specified address, otherwise you will get the following error:
Operating system error code 5(Access is denied.)
FROM '\\SERVER\\FOLDER\\FOLDER\\FileName.txt'
That's it.
Sorry.THX.

Powershell Script to select records in PostgreSQL database [duplicate]

I'm trying to connect to a remote PostgreSql database using powershell. This is my first time using powershell so I'm sorry if this is a noob question. This is my Code:
$DBConnectionString = "Driver={PostgreSQL UNICODE}:Server=$MyServer;Port=$MyPort;Database=$MyDB;Uid=$MyUid;Pwd=$MyPass;"
$DBConn = New-Object System.Data.Odbc.OdbcConnection;
$DBConn.ConnectionString = $DBConnectionString;
$DBConn.Open();
$DBCmd = $DBConn.CreateCommand();
$DBCmd.CommandText = "SELECT * FROM mytable;";
$DBCmd.ExecuteReader();
$DBConn.Close();
When I run this I get "Exception Calling "Open" with "0" argument(s): ERROR [IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified". I've downloaded and installed the pgsqlodbc driver but I'm still getting this error. Does anyone have any ideas how I could fix this? I have searched the internet and I'm really not getting anywhere at this point.
Thanks.
Consult: https://odbc.postgresql.org/
Download: https://www.postgresql.org/ftp/odbc/versions/msi/
Data sources (ODBC) on Windows: Start → Search → odbc → User DSN → Add/Configure
Example :
$MyServer = "<ip>"
$MyPort = "5432"
$MyDB = "<database>"
$MyUid = "<user>"
$MyPass = "<pass>"
$DBConnectionString = "Driver={PostgreSQL UNICODE(x64)};Server=$MyServer;Port=$MyPort;Database=$MyDB;Uid=$MyUid;Pwd=$MyPass;"
$DBConn = New-Object System.Data.Odbc.OdbcConnection;
$DBConn.ConnectionString = $DBConnectionString;
$DBConn.Open();
$DBCmd = $DBConn.CreateCommand();
$DBCmd.CommandText = "SELECT * FROM tb_module;";
$DBCmd.ExecuteReader();
$DBConn.Close();
You can use psql which comes with postgresql if you happen to have postgresql installed on your client
$dburl="postgresql://exusername:expw#exhostname:5432/postgres"
$data="select * from extable" | psql --csv $dburl | ConvertFrom-Csv
You must have psql in your path or reference it, its within e.g. C:\Program Files\PostgreSQL\12\bin. Should be able to type "psql" and see output within powershell.
Check if the DSN exists in ODBC data source. If not you have to create one going to 'Control Panel', 'Admin. Tools', 'Data Sources (ODBC)'. Then select 'Add User DSN'-
Select the PostgreSQL driver, and fill in your server and database details.
Test connection to check is all ok!
You actually have a typo in your connection string after Driver declaration.
There is a double colon instead of a semicolon :)
Wrong:
{PostgreSQL UNICODE} : Server
Correct:
{PostgreSQL UNICODE} ; Server
I found the problem, I thought the Postgresql ODBC driver was installed, but it wasn't. I finally got it to work after finding this site: http://code.google.com/p/visionmap/wiki/psqlODBC Then I followed the instructions above. it works.
Thanks for all the help.

Connect to remote PostgreSql database using Powershell

I'm trying to connect to a remote PostgreSql database using powershell. This is my first time using powershell so I'm sorry if this is a noob question. This is my Code:
$DBConnectionString = "Driver={PostgreSQL UNICODE}:Server=$MyServer;Port=$MyPort;Database=$MyDB;Uid=$MyUid;Pwd=$MyPass;"
$DBConn = New-Object System.Data.Odbc.OdbcConnection;
$DBConn.ConnectionString = $DBConnectionString;
$DBConn.Open();
$DBCmd = $DBConn.CreateCommand();
$DBCmd.CommandText = "SELECT * FROM mytable;";
$DBCmd.ExecuteReader();
$DBConn.Close();
When I run this I get "Exception Calling "Open" with "0" argument(s): ERROR [IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified". I've downloaded and installed the pgsqlodbc driver but I'm still getting this error. Does anyone have any ideas how I could fix this? I have searched the internet and I'm really not getting anywhere at this point.
Thanks.
Consult: https://odbc.postgresql.org/
Download: https://www.postgresql.org/ftp/odbc/versions/msi/
Data sources (ODBC) on Windows: Start → Search → odbc → User DSN → Add/Configure
Example :
$MyServer = "<ip>"
$MyPort = "5432"
$MyDB = "<database>"
$MyUid = "<user>"
$MyPass = "<pass>"
$DBConnectionString = "Driver={PostgreSQL UNICODE(x64)};Server=$MyServer;Port=$MyPort;Database=$MyDB;Uid=$MyUid;Pwd=$MyPass;"
$DBConn = New-Object System.Data.Odbc.OdbcConnection;
$DBConn.ConnectionString = $DBConnectionString;
$DBConn.Open();
$DBCmd = $DBConn.CreateCommand();
$DBCmd.CommandText = "SELECT * FROM tb_module;";
$DBCmd.ExecuteReader();
$DBConn.Close();
You can use psql which comes with postgresql if you happen to have postgresql installed on your client
$dburl="postgresql://exusername:expw#exhostname:5432/postgres"
$data="select * from extable" | psql --csv $dburl | ConvertFrom-Csv
You must have psql in your path or reference it, its within e.g. C:\Program Files\PostgreSQL\12\bin. Should be able to type "psql" and see output within powershell.
Check if the DSN exists in ODBC data source. If not you have to create one going to 'Control Panel', 'Admin. Tools', 'Data Sources (ODBC)'. Then select 'Add User DSN'-
Select the PostgreSQL driver, and fill in your server and database details.
Test connection to check is all ok!
You actually have a typo in your connection string after Driver declaration.
There is a double colon instead of a semicolon :)
Wrong:
{PostgreSQL UNICODE} : Server
Correct:
{PostgreSQL UNICODE} ; Server
I found the problem, I thought the Postgresql ODBC driver was installed, but it wasn't. I finally got it to work after finding this site: http://code.google.com/p/visionmap/wiki/psqlODBC Then I followed the instructions above. it works.
Thanks for all the help.

PostgreSQL pgp_sym_encrypt() broken in version 9.1

The following works in PostgreSQL 8.4:
insert into credentials values('demo', pgp_sym_encrypt('password', 'longpassword'));
When I try it in version 9.1 I get this:
ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist LINE
1: insert into credentials values('demo', pgp_sym_encrypt('pass...
^ HINT: No function matches the given name and argument types. You might need to add
explicit type casts.
*** Error ***
ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist SQL
state: 42883 Hint: No function matches the given name and argument
types. You might need to add explicit type casts. Character: 40
If I try some explicit casts like this
insert into credentials values('demo', pgp_sym_encrypt(cast('password' as text), cast('longpassword' as text)))
I get a slightly different error message:
ERROR: function pgp_sym_encrypt(text, text) does not exist
I have pgcrypto installed. Does anyone have pgp_sym_encrypt() working in PostgreSQL 9.1?
On explanation could be that the module was installed into a schema that is not in your search path - or to the wrong database.
Diagnose your problem with this query and report back the output:
SELECT n.nspname, p.proname, pg_catalog.pg_get_function_arguments(p.oid) as params
FROM pg_catalog.pg_proc p
JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace
WHERE p.proname ~~* '%pgp_sym_encrypt%'
AND pg_catalog.pg_function_is_visible(p.oid);
Finds functions in all schemas in your database. Similar to the psql meta-command
\df *pgp_sym_encrypt*
Make sure you install the extension on the desired schema.
sudo -i -u postgres
psql $database
CREATE EXTENSION pgcrypto;
OK, problem solved.
I was creating the pgcrypto extension as the first operation in the script. Then I dropped and added the VGDB database. That's why pgcrypto was there immediately after creating it, but didn't exist when running the sql later in the script or when I opened pgadmin.
This script is meant for setting up new databases and if I had tried it on a new database the create extension would have failed right away.
My bad. Thanks for the help, Erwin.
Just mention de schema where is installed pgcrypto like this:
#ColumnTransformer(forColumn = "TEST",
read = "public.pgp_sym_decrypt(TEST, 'password')",
write = "public.pgp_sym_encrypt(?, 'password')")
#Column(name = "TEST", columnDefinition = "bytea", nullable = false)
private String test;
I ran my (python) script again and the CREATE EXTENSION ran without error. The script also executes this command
psql -d VGDB -U postgres -c "select * from pg_available_extensions order by name"
which includes the following in the result set:
pgcrypto | 1.0 | 1.0 | cryptographic functions
So psql believes that it has installed pgcrypto.
Later in the same script when I execute
psql -d VGDB -U postgres -f sql/Create.Credentials.table.sql
where sql/Create.Credentials.table.sql includes this
insert into credentials values('demo', pgp_sym_encrypt('password', 'longpassword'));
I get this
psql:sql/Create.Credentials.table.sql:31: ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist
LINE 1: insert into credentials values('demo', pgp_sym_encrypt('pass...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
When I open pgadmin it does not show pgcrypto in either the VGDB or postgres databases even though the query above called by psql shows that pgcrypto is installed.
Could there be an issue with needing to commit after using psql to execute the "create extension ..." command? None of my other DDL or SQL statements require a commit when they get executed with psql.
It's starting to look like psql is just flakey. Is there another way to call "create extension pgcrypto" - e.g. with Python's database support classes - or does that have to be run through psql?