I am working with a script made for SQLite but I need to adapt it to a PostgreSQL database and this line is always returning an error:
PRAGMA table_info(table)
How can I get an equivalent result on postgres >
Found it !
SELECT *
FROM information_schema.columns
WHERE table_name = 'table_name'
I am trying to run a sqoop job to ingest data from postgresql to hdfs and I am stuck on some point.
Sqoop adds " AND (1=0)" to the end of my "WHERE" statement to fetch metadata just before ingestion.
sqoop import
--connect jdbc:postgresql://randomtexthere.com:5432/test
--username user
-P
--query
"
SELECT *
FROM table1 pr
INNER JOIN
table2 fr
ON pr.id = fr.id
WHERE fr.another_id > 12345 AND fr.another_id < 123456 AND \$CONDITIONS
"
--hcatalog-database test
--hcatalog-storage-stanza "STORED AS PARQUET"
--hcatalog-table table1--split-by id
Once above command is added, query never gets completed. (both in Sqoop and DBeaver)
However this query only works once I set SET OPTIMIZER = ON (in DBeaver)
SET OPTIMIZER = ON;
SELECT *
FROM table1 pr
INNER JOIN
table2 fr
ON pr.id = fr.id
WHERE fr.another_id > 12345 AND fr.another_id < 123456 AND (1=0);
I am looking for a solution to set optimizer parameter in my sqoop session.
Is there a way to do it?
passing the optimizer hint in jdbc connection solves the issue.
--connect jdbc:postgresql://randomtexthere.com:5432/test?optimizer=true
I'm importing shape files into Postgresql via this command:
ogr2ogr PG:host=localhost dbname=someDbName user=someUserName password=somePassword shapeFile.shp -nln alternateLayerName -nlt someValidGeometry
This works well but goes into public schema in Postgresql. I'd like to choose a different schema. Is there a way to achieve this with ogr2ogr alone?
There's no reference in man ogr2ogr to schema. My web search wasn't fruitful either.
I know I can do an ALTER TABLE some_table set schema a_different_schema, but that would mean adding another step to the process.
$ ogr2ogr --version
GDAL 2.1.0, released 2016/04/25
After a better web search I found on GDAL pg driver documentation that one can use ACTIVE_SCHEMA=string: Active schema. to set the schema where the table will be created.
I tried it like this:
ogr2ogr -f "PostgreSQL" PG:"dbname=mydb active_schema=layers" country.shp -nln test_table -nlt MULTILINESTRING
And it complains with:
ERROR 1: PQconnectdb failed: invalid connection option "active_schema"
But table gets created and populated properly. So I guess it's ok.
I found a better solution by using lco option:
ogr2ogr -f PostgreSQL "PG:host=localhost port=5432 dbname=some_db
user=postgres password=" someShapeFile.shp -nln desiredTableName
-nlt someValidGeometry -lco SCHEMA=desiredPostgresqlSchema
I open a command prompt in Ubuntu and then log in to Firebird, like so:
$ isql-fb
SQL> connect "localhost:/var/lib/firebird/2.5/data/reestr.fdb" user 'SYSDBA' password 'root';
Then I list all tables in my database:
> show tables;
ARCHIVE_1_ ...
...
...
Finally, I want to drop one table. I try it this way:
> DROP TABLE ARCHIVE_1_;
........ absolutely no reaction, propmt is waiting for something
If I log in again and list tables, I see that the table is still there. So, what is wrong with all that?
EDIT
This is what set; command in isql prompt returns:
Print statistics: OFF
Echo commands: OFF
List format: OFF
List Row Count: OFF
Select rowcount limit: 0
Autocommit DDL: ON
Access Plan: OFF
Access Plan only: OFF
Display BLOB type: 1
Column headings: ON
Terminator: ;
Time: OFF
Warnings: ON
Bail on error: OFF
It could be that you have turned off autocommit of DDL statements (the default is on). To check use set; command in the isql, it'll list the current setup. If the autoddl is off then you can set it on again using SET AUTODDL ON; or just issue commit; after you'r DROP statement.
The following works in PostgreSQL 8.4:
insert into credentials values('demo', pgp_sym_encrypt('password', 'longpassword'));
When I try it in version 9.1 I get this:
ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist LINE
1: insert into credentials values('demo', pgp_sym_encrypt('pass...
^ HINT: No function matches the given name and argument types. You might need to add
explicit type casts.
*** Error ***
ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist SQL
state: 42883 Hint: No function matches the given name and argument
types. You might need to add explicit type casts. Character: 40
If I try some explicit casts like this
insert into credentials values('demo', pgp_sym_encrypt(cast('password' as text), cast('longpassword' as text)))
I get a slightly different error message:
ERROR: function pgp_sym_encrypt(text, text) does not exist
I have pgcrypto installed. Does anyone have pgp_sym_encrypt() working in PostgreSQL 9.1?
On explanation could be that the module was installed into a schema that is not in your search path - or to the wrong database.
Diagnose your problem with this query and report back the output:
SELECT n.nspname, p.proname, pg_catalog.pg_get_function_arguments(p.oid) as params
FROM pg_catalog.pg_proc p
JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace
WHERE p.proname ~~* '%pgp_sym_encrypt%'
AND pg_catalog.pg_function_is_visible(p.oid);
Finds functions in all schemas in your database. Similar to the psql meta-command
\df *pgp_sym_encrypt*
Make sure you install the extension on the desired schema.
sudo -i -u postgres
psql $database
CREATE EXTENSION pgcrypto;
OK, problem solved.
I was creating the pgcrypto extension as the first operation in the script. Then I dropped and added the VGDB database. That's why pgcrypto was there immediately after creating it, but didn't exist when running the sql later in the script or when I opened pgadmin.
This script is meant for setting up new databases and if I had tried it on a new database the create extension would have failed right away.
My bad. Thanks for the help, Erwin.
Just mention de schema where is installed pgcrypto like this:
#ColumnTransformer(forColumn = "TEST",
read = "public.pgp_sym_decrypt(TEST, 'password')",
write = "public.pgp_sym_encrypt(?, 'password')")
#Column(name = "TEST", columnDefinition = "bytea", nullable = false)
private String test;
I ran my (python) script again and the CREATE EXTENSION ran without error. The script also executes this command
psql -d VGDB -U postgres -c "select * from pg_available_extensions order by name"
which includes the following in the result set:
pgcrypto | 1.0 | 1.0 | cryptographic functions
So psql believes that it has installed pgcrypto.
Later in the same script when I execute
psql -d VGDB -U postgres -f sql/Create.Credentials.table.sql
where sql/Create.Credentials.table.sql includes this
insert into credentials values('demo', pgp_sym_encrypt('password', 'longpassword'));
I get this
psql:sql/Create.Credentials.table.sql:31: ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist
LINE 1: insert into credentials values('demo', pgp_sym_encrypt('pass...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
When I open pgadmin it does not show pgcrypto in either the VGDB or postgres databases even though the query above called by psql shows that pgcrypto is installed.
Could there be an issue with needing to commit after using psql to execute the "create extension ..." command? None of my other DDL or SQL statements require a commit when they get executed with psql.
It's starting to look like psql is just flakey. Is there another way to call "create extension pgcrypto" - e.g. with Python's database support classes - or does that have to be run through psql?