Get PostgreSQL execution plan with Drill query - postgresql

I need to get the execution plan of a query that execute from Apache Drill using the PostgreSQL storage plugin (not the Drill execution plan, but the PostGIS one).
So I enable the explain plan logs with following commands;
SET auto_explain.log_min_duration = 0;
SET auto_explain.log_analyze = true;
And if I execute a query from pgAdmin, it shows the statement and the plan. But if I execute the same query from Drill, it does not log anything.
Do you know why this happens and how can be solved this situation?
Note: I checked the connection and it's ok, they are the same in pgAdmin and Drill, also in Drill I execute queries and I get results, so I assume that there is no connection problems.

I suspect you are executing the SET commands from a postgres command line, so those settings would only apply to Drill's postgres session. In order to apply those settings to Drill's postgres session, try adding those properties to Drill's storage plugin configuration. Here is an example configuration for those properties:
{
"type": "jdbc",
"driver": "org.postgresql.Driver",
"url": "jdbc:postgresql://localhost:5432;auto_explain.log_min_duration=0;auto_explain.log_analyze=true",
"username": "postgres",
"enabled": true
}

Related

How to log all DDLs on a gcloud SQL instance?

I would like to log all DDL queries that are run on SQL instance. I tried looking into plugins but they are not allowed.
Edit: It's a MySQL instance.
You may set the Cloud SQL Flag "log_statement" to value "mod" to log all Data definition language (DDL) statements.
The reference link :
https://cloud.google.com/sql/docs/postgres/flags#postgres-l

Explicitly set user schema in Bookshelf for PG

In our dev environment, each dev has their own schema to replicate the stage and prod databases. Need to be able to pass user schema as part of the client config. Here's the config...
"client": {
"username": "user_login",
"password": "user_password",
"host": "db_host",
"port": 5432,
"database": "db_name",
"dialect": "pg",
"schema": "user_schema"
}
...and here's the instantiation...
var Knex = require('knex')({
client: client.dialect,
connection: {
host: client.host,
user: client.username,
password: client.password,
database: client.database,
searchPath: client.schema
});
var Bookshelf = require('bookshelf')(Knex);
Inspecting the Bookshelf and Knex objects, I can't determine anywhere which schema is being used. I had to dig into the SchemaCompiler_PG.prototype.hasTable method, where I happened to find it in an array on a query response object, which includes a property of table_schema, happily set to public. I tried to set it to my schema, but nothing succeeded.
So we set default search paths for each user login.
ALTER ROLE user_login SET search_path TO 'user_schema'
Inspecting the response object, table_schema was set to my search path. But when another dev tried to run the code on her machine (local instances of Node and PG), that response object was set to MY schema. We can find no way to manually set it.
No idea what kind of black magic that would be, but need to know how to set the user schema for Bookshelf/Knex/PG.
While I haven't been able to find an answer for the original question, how to set the PG schema, I was able to resolve the issue at hand. Turned out to be a permissions issue in PG. Ensure that all users had r/w perms in their respective search_path and all was well.
No error messages that I could find anywhere within Bookshelf, Knex or PG client pointed to this as a possible culprit, so hopefully this helps somebody.

ERROR: cannot execute CREATE TABLE in a read-only transaction

I'm trying to setup the pgexercises data in my local machine. When I run: psql -U <username> -f clubdata.sql -d postgres -x I get the error: psql:clubdata.sql:6: ERROR: cannot execute CREATE SCHEMA in a read-only transaction.
Why did it create a read-only database on my local machine? Can I change this?
Normally the most plausible reasons for this kind of error are :
trying create statements on a read-only replica (the entire instance is read-only).
<username> has default_transaction_read_only set to ON
the database has default_transaction_read_only set to ON
The script mentioned has in its first lines:
CREATE DATABASE exercises;
\c exercises
CREATE SCHEMA cd;
and you report that the error happens with CREATE SCHEMA at line 6, not before.
That means that the CREATE DATABASE does work, when run by <username>.
And it wouldn't work if any of the reasons above was directly applicable.
One possibility that would technically explain this would be that default_transaction_read_only would be ON in the postgresql.conf file, and set to OFF for the database postgres, the one that the invocation of psql connects to, through an ALTER DATABASE statement that supersedes the configuration file.
That would be why CREATE DATABASE works, but then as soon as it connects to a different database with \c, the default_transaction_read_only setting of the session would flip to ON.
But of course that would be a pretty weird and unusual configuration.
Reached out to pgexercises.com and they were able to help me.
I ran these commands(separately):
psql -U <username> -d postgres
begin;
set transaction read write;
alter database exercises set default_transaction_read_only = off;
commit;
\q
Then I dropped the database from the terminal dropdb exercises and ran script again psql -U <username> -f clubdata.sql -d postgres -x -q
I was having getting cannot execute CREATE TABLE in a read-only transaction, cannot execute DELETE TABLE in a read-only transaction and others.
They all followed a cannot execute INSERT in a read-only transaction. It was like the connection had switched itself over to read-only in the middle of my batch processing.
Turns out, I was running out of storage!
Write access was disabled when the database could no longer write anything. I am using Postgres on Azure. I don't know if the same effect would happen if I was on a dedicated server.
I had same issue for Postgre Update statement
SQL Error: 0, SQLState: 25006 ERROR: cannot execute UPDATE in a read-only transaction
Verified Database access by running below query and it will return either true or false
SELECT pg_is_in_recovery()
true -> Database has only Read Access
false -> Database has full Access
if returns true then check with DBA team for the full access and also try for ping in command prompt and ensure the connectivity.
ping <database hostname or dns>
Also verify if you have primary and standby node for the database
In my case I had a master and replication nodes, and the master node became replication node, which I believe switched it into hot_standby mode. So I was trying to write data into a node that was meant only for reading, therefore the "read-only" problem.
You can query the node in question with SELECT pg_is_in_recovery(), and if it returns True then it is "read-only", and I suppose you should switch to using whatever master node you have now.
I got this information from: https://serverfault.com/questions/630753/how-to-change-postgresql-database-from-read-only-to-writable.
So full credit and my thanks goes to Craig Ringer!
Dbeaver: In my case
This was on.
This doesn't quite answer the original question, but I received the same error and found this page, which ultimately led to a fix.
My issue was trying to run a function with temp tables being created and dropped. The function was created with SECURITY DEFINER privileges, and the user had access locally.
In a different environment, I received the cannot execute DROP TABLE in a read-only transaction error message. This environment was AWS Aurora, and by default, non-admin developers were given read-only privileges. Their server connections were thus set up to use the read-only node of Aurora (-ro- is in the connection url), which must put the connection in the read-only state. Running the same function with the same user against the write node worked.
Seems like a good use case for table variables like SQL Server has! Or, at least, AWS should modify their flow to allow temp tables to be created and dropped on read nodes.
This occurred when I was restoring a production database locally, the database is still doing online recovery from the WAL records.
A little bit unexpected as I assumed pgbackgrest was creating instantly recoverable restores, perhaps not.
91902 postgres 20 0 1445256 14804 13180 D 4.3 0.3 0:28.06 postgres: startup recovering 000000010000001E000000A5
If like me you are trying to create DB on heroku and are stuck as this message shows up on the dataclip tab
I did this,
Choose Resources from(Overview Resources Deploy Metrics Activity Access Settings)
Choose Settings out of (Overview, Durability, Settings, Dataclip)
Then in Administration->Database Credentials choose View Credentials...
then open terminal and fill that info here and enter
psql --host=***************.amazonaws.com --port=5432 --username=*********pubxl --password --dbname=*******lol
then it'll ask for password, copy-paste from there and you can run Postgres cmds.
I suddenly started facing this error on postgres installed on my windows machine, when I was running alter query from dbeaver, all I did was deleted the connection of postgres from dbeaver and created a new connection
If you are using Azure Database for PostgreSQL your server gets into read-only mode when the storage used is near total capacity.
The error you get is exactly:
ERROR: cannot execute XXXXXXXXX in a read-only transaction
https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-compute-storage
I just had this error. My cause was not granting permission to the SEQUENCE
GRANT ALL ON SEQUENCE word_mash_word_cube_template_description_reference_seq TO ronshome_user;
If you are facing this issue with an RDS instance cluster, please check your endpoint and use the Writer instance endpoint. Then it should work now.
Issue can be dur to Intellij config:
Go to Database view> click on Data Source Properties (Shift + enter)> (Select your data source)>
Options tab> Under Connection : uncheck Read-only
For me it was Azure PostgreSQL failing over to standby during maintaince in Azure and never failing back to master when PostgreSQL was in HA mode. You can check this event in Service Health and also check which zone you current VM is running from. If it's 2 and not 1 them most likely that's the result of events described above.

App to monitor PostgreSQL queries in real time?

I'd like to monitor the queries getting sent to my database from an application. To that end, I've found pg_stat_activity, but more often then not, the rows which are returned read " in transaction". I'm either doing something wrong, am not fast enough to see the queries come through, am confused, or all of the above!
Can someone recommend the most idiot-proof way to monitor queries running against PostgreSQL? I'd prefer some sort of easy-to-use UI based solution (example: SQL Server's "Profiler"), but I'm not too choosy.
PgAdmin offers a pretty easy-to-use tool called server monitor
(Tools ->ServerStatus)
With PostgreSQL 8.4 or higher you can use the contrib module pg_stat_statements to gather query execution statistics of the database server.
Run the SQL script of this contrib module pg_stat_statements.sql (on ubuntu it can be found in /usr/share/postgresql/<version>/contrib) in your database and add this sample configuration to your postgresql.conf (requires re-start):
custom_variable_classes = 'pg_stat_statements'
pg_stat_statements.max = 1000
pg_stat_statements.track = top # top,all,none
pg_stat_statements.save = off
To see what queries are executed in real time you might want to just configure the server log to show all queries or queries with a minimum execution time. To do so set the logging configuration parameters log_statement and log_min_duration_statement in your postgresql.conf accordingly.
pg_activity is what we use.
https://github.com/dalibo/pg_activity
It's a great tool with a top-like interface.
You can install and run it on Ubuntu 21.10 with:
sudo apt install pg-activity
pg_activity
If you are using Docker Compose, you can add this line to your docker-compose.yaml file:
command: ["postgres", "-c", "log_statement=all"]
now you can see postgres query logs in docker-compose logs with
docker-compose logs -f
or if you want to see only postgres logs
docker-compose logs -f [postgres-service-name]
https://stackoverflow.com/a/58806511/10053470
I haven't tried it myself unfortunately, but I think that pgFouine can show you some statistics.
Although, it seems it does not show you queries in real time, but rather generates a report of queries afterwards, perhaps it still satisfies your demand?
You can take a look at
http://pgfouine.projects.postgresql.org/

Oracle SQL Developer and PostgreSQL

I'm trying to connect to a PostgreSQL 9.1 database using Oracle SQL Developer 3.0.04, but I'm not having any success so far.
First, if I add a third party driver on preferences, when adding a new connection there's no tab for PostgreSQL (it works fine for MySQL though). I'm using a JDBC4 version 9.1 driver, but I tried a JDBC3 of the same version and still get the same thing.
Second, there's nothing like manual configuration tab when adding a new connection. The closest is the Advanced option on Oracle tab, where I can provide a custom URL, but it fails because complains about the selected Driver (of course).
Finally, I got connected importing a connection from an XML file (contents below), but it displays only my schemas and doesn't show my tables inside them.
So, my question is: does Orable SQL Developer support PostgreSQL connections? Is there any other way to have my tables being displayed in the ObjectViewer?
<?xml version = '1.0' encoding = 'UTF-8'?>
<References xmlns="http://xmlns.oracle.com/adf/jndi">
<Reference name="Lumea" className="oracle.jdeveloper.db.adapter.DatabaseProvider" credentialStoreKey="Lumea" xmlns="">
<Factory className="oracle.jdeveloper.db.adapter.DatabaseProviderFactory"/>
<RefAddresses>
<StringRefAddr addrType="user">
<Contents>lumea</Contents>
</StringRefAddr>
<StringRefAddr addrType="subtype">
<Contents>thirdParty</Contents>
</StringRefAddr>
<StringRefAddr addrType="customUrl">
<Contents>jdbc:postgresql://localhost:5432/versates</Contents>
</StringRefAddr>
<StringRefAddr addrType="SavePassword">
<Contents>true</Contents>
</StringRefAddr>
<StringRefAddr addrType="password">
<Contents>myencryptedpass</Contents>
</StringRefAddr>
<StringRefAddr addrType="driver">
<Contents>org.postgresql.Driver</Contents>
</StringRefAddr>
<StringRefAddr addrType="DeployPassword">
<Contents>true</Contents>
</StringRefAddr>
</RefAddresses>
</Reference>
</References>
Oracle SQL Developer 4.0.1.14 surely does support connections to PostgreSQL.
download JDBC driver for Postgres (http://jdbc.postgresql.org/download.html)
in SQL Developer go to Tools → Preferences, Database → Third Party JDBC Drivers and add the jar file (see http://www.oracle.com/technetwork/products/migration/jdbc-migration-1923524.html for step by step example)
now just make a new Database Connection and instead of Oracle, select PostgreSQL tab
Edit:
If you have different user name and database name, one should specify in hostname: hostname/database? (do not forget ?) or hostname:port/database?.
(thanks to #kinkajou and #Kloe2378231; more details on https://stackoverflow.com/a/28671213/565525).
I've just downloaded SQL Developer 4.0 for OS X (10.9), it just got out of beta. I also downloaded the latest Postgres JDBC jar. On a lark I decided to install it (same method as other third party db drivers in SQL Dev), and it accepted it. Whenever I click "new connection", there is a tab now for Postgres... and clicking it shows a panel that asks for the database connection details.
The answer to this question has changed, whether or not it is supported, it seems to work. There is a "choose database" button, that if clicked, gives you a dropdown list filled with available postgres databases. You create the connection, open it, and it lists the schemas in that database. Most postgres commands seem to work, though no psql commands (\list, etc).
Those who need a single tool to connect to multiple database engines can now use SQL Developer.
Oracle SQL Developer doesn't support connections to PostgreSQL. Use pgAdmin to connect to PostgreSQL instead, you can get it from the following URL
http://www.pgadmin.org/download/windows.php
I managed to connect to postgres with SQL Developer.
I downloaded postgres jdbc driver and saved it in a folder.
I run SQL Developer -> Tools -> Preferences -> Search -> JDBC
I defined postgres jdbc driver for the Database and Data Modeler (optional).
This is the trick.
When creating new connection define Hostname like localhost/crm? where crm is the database name. Test the connection, works fine.
If there is no database with the same name as the username, then clicking "Choose Database" will fail with an error like "Status : Failure -FATAL: database "your_username" does not exist"
To work around this, put 5432/database_name? in the Port field, where 5432 is the port of your Postgres instance and database_name is the name of at an existing database that your_username has access to. Then click "Choose Database" again and it should work. Now you can choose the database you want and remove the extra /database_name? from the Port field.
I think this question needs an updated answer, because both PostgreSQL and SQLDeveloper have been updated several times since it was originally asked.
I've got a PostgreSQL instance running in Azure, with SSLMODE=Require.
While I've been using DBeaverCE to access that instance and generate an ER Diagram, I've gotten really familiar with SQLDeveloper, which is now at v19.4.
The instructions about downloading the latest PostgreSQL JDBC driver and where to place it are correct. What has changed, though, is where to configure your DB access.
You'll find a file $HOME/.sqldeveloper/system19.4.0.354.1759/o.jdeveloper.db.connection.19.3.0.354.1759/connections.json:
{
"connections": [
{
"name": "connection-name-goes-here",
"type": "jdbc",
"info": {
"customUrl": "jdbc:postgresql://your-postgresql-host:5432/DBNAME?sslmode=require",
"hostname": "your-postgresql-host",
"driver": "org.postgresql.Driver",
"subtype": "SDPostgreSQL",
"port": "5432",
"SavePassword": "false",
"RaptorConnectionType": "SDPostgreSQL",
"user": "your_admin_user",
"sslmode": "require"
}
}
]
}
You can use this connection with both Data Modeller and the admin functionality of SQLDeveloper. Specifying all the port, dbname and sslmode in the customUrl are required because SQLDeveloper isn't including the sslmode in what it sends via JDBC, so you have to construct it by hand.
I got the list of databases to populate by putting my username in the Username field (no password) and clicking "Choose Database". Doesn't work with a blank Username field, I can only connect to my user database that way.
(This was with SQL Developer 4.0.0.13, Postgres.app 9.3.0.0, and postgresql-9.3-1100.jdbc41.jar, FWIW.)
Oracle SQL Developer 2020-02 support PostgreSQL, but it is just the basics by adding postgreSQL driver under jdbc dir and configure by adding as a 3rd party driver.
The supported functionality:
multiple databases which can be selected at connection definition
CRUD operations like query tables
scheme operations
basic modelling support: show tables without pk, fk, connections
Not supported functionalities:
no table or field completion
no indexes are shown in a tab
no constraints are shown in a tab like: fk, pk-s, unique, or others
no table or field completions in the editor
no functions, packages,triggers, views are shown
The sad thing is Oracle should only change the queries behind this view in case of PostgreSql connections. For example for indexes they need to use this query: select * from pg_catalog.pg_indexes;
Except that it will not work if your user name and database name are differents. It sounds like an SQLDeveloper bug and i can't find any workaroud
Maybe there are some bugs in Oracle SQL Developer when it connect to the postgresql.However I connect postgresql with navicat successfully.(My postgresql username and database name are different
host= localhost/postgres? worked for me if you need a schema/database use:
localhost/postgres?currentSchema=myschema
ex: localhost/postgres?currentSchema=public
once connected, you can also use the
[Choose Database] button and pull down
sql developer postgres jdbc url for the rest of us