Web2py Scheduler is creating "Idle in Transaction" process with PostgreSQL - postgresql

This question is specific to scheduler in Web2py Framework.
I am using scheduler as a Linux Service.
I created a file /etc/systemd/system/web2py-sched.service
[Unit]
Description=Web2Py scheduler service
[Service]
ExecStart=/usr/bin/python /home/www-data/web2py/web2py.py -K myApp
Type=simple
Restart=always
[Install]
WantedBy=multi-user.target
Then install the service calling:
sudo systemctl enable /etc/systemd/system/web2py-sched.service
systemctl start /etc/systemd/system/web2py-sched
When I checked my Postgres status, I see there are processes with "Idle in Transaction" status.
may#myApp:/home/www-data/web2py/scmyAppts$ sudo systemctl status postgresql#9.5-main
● postgresql#9.5-main.service - PostgreSQL Cluster 9.5-main
Loaded: loaded (/lib/systemd/system/postgresql#.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2018-10-16 15:39:25 PDT; 1 weeks 1 days ago
Main PID: 28202 (postgres)
CGroup: /system.slice/system-postgresql.slice/postgresql#9.5-main.service
├─10956 postgres: may mydb ::1(60654) idle in transaction
├─10957 postgres: may mydb ::1(60656) idle in transaction
├─13421 postgres: may mydb 10.5.77.35(51998) idle
├─13422 postgres: may mydb 10.5.77.35(52004) idle
├─13423 postgres: may postgres 10.5.77.35(52005) idle
├─13425 postgres: may mydb 10.5.77.35(52007) idle
├─13433 postgres: may mydb 10.5.77.35(52332) idle
├─28202 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
├─28204 postgres: checkpointer process
├─28205 postgres: writer process
├─28206 postgres: wal writer process
├─28207 postgres: autovacuum launcher process
├─28208 postgres: stats collector process
├─30475 postgres: may mydb ::1(59838) idle
├─30476 postgres: may mydb ::1(59840) idle in transaction
└─30500 postgres: may mydb ::1(59842) idle
When I observed few of process are idle and transaction and my DB Queries are timing out, I checked to see which table is acquiring locks.
SELECT relation, pid, mode, granted, relname
FROM pg_locks
INNER JOIN pg_stat_user_tables
ON pg_locks.relation = pg_stat_user_tables.relid
WHERE pg_locks.pid='30476';
relation pid mode granted relname
16553 30476 AccessShareLock TRUE scheduler_task
Please help me on this, I have thoroughly gone thru my scheduler code and there is db.commit() called after every transaction.
Version:
Version
2.17.2-stable+timestamp.2018.10.06.18.54.02
(Running on nginx/1.10.3, Python 2.7.12)
web2py is up to date

Is your app really called 'myApp' as you configured by the service start ExecStart=/usr/bin/python /home/www-data/web2py/web2py.py -K myApp?
You have to specify your application by name. 'myapp' is just a placeholder.
Best regards Clemens

Related

Postgres on Ubuntu, controlling the postgresql.conf location

I created a cluster as follows:
pg_createcluster -d /some_dir/pg_data/ -p 5432 \
--environment=/some_dir/pg_env.prod.conf \
--createclusterconf=/some_dir/pg.prod.conf \
13 apif
However, when the cluster is stood-up, it has effectively copied the pg.prod.conf file to the default directory (-c config_file=/etc/postgresql/13/apif/postgresql.conf) rather than using it. That means if I make changes to the file, of course, they won't be seen. Thats not what I wanted.
sudo systemctl status postgresql#13-apif
● postgresql#13-apif.service - PostgreSQL Cluster 13-apif
Loaded: loaded (/lib/systemd/system/postgresql#.service; indirect; vendor preset: enabled)
Active: active (running) since Wed 2021-12-15 21:15:25 UTC; 6s ago
Process: 6791 ExecStop=/usr/bin/pg_ctlcluster --skip-systemctl-redirect -m fast 13-apif stop (code=exited, status=2)
Process: 6656 ExecReload=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 13-apif reload (code=exited, status=0/SUCCESS)
Process: 6868 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 13-apif start (code=exited, status=0/SUCCESS)
Main PID: 6873 (postgres)
Tasks: 9 (limit: 4915)
CGroup: /system.slice/system-postgresql.slice/postgresql#13-apif.service
├─6873 /usr/lib/postgresql/13/bin/postgres -D /some_dir/apif -c config_file=/etc/postgresql/13/apif/postgresql.conf
├─6874 postgres: logger
├─6876 postgres: checkpointer
├─6877 postgres: background writer
├─6878 postgres: walwriter
├─6879 postgres: autovacuum launcher
├─6880 postgres: stats collector
├─6881 postgres: pg_cron launcher
└─6882 postgres: logical replication launcher
Dec 15 21:15:22 ip-172-33-2-19 systemd[1]: Starting PostgreSQL Cluster 13-apif...
Dec 15 21:15:25 ip-172-33-2-19 systemd[1]: Started PostgreSQL Cluster 13-apif.
...
How do I do this the way I want to?... where the cluster is configured to reference a file of my choosing, permanently.
I tried to find an answer and couldn't.
But I did find an acceptable work-around. And while this isn't quite what I was looking for because it doesn't explicitly specify the location of the conf file, it does allow me to reference it in a replicable one-liner.
pg_createcluster -d /some_dir/pg_data/ -p 5432 \
--environment=/some_dir/pg_env.prod.conf \
--pgoption include_if_exists=/some_dir/pg.prod.conf \
13 apif

systemctl status postgresql.service showing output as differently

Not coming out of os(Linux) prompt.Need to do ctrl+c to come out from prompt
output is not showing as expected(Active: activating (start) instead of Active: active (running))
but postgresql services are starting.
# systemctl status postgresql.service
● postgresql.service - PostgreSQL database server
Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: disabled)
Active: activating (start) since Thu 2021-02-04 12:38:17 CET; 15min ago
Docs: man:postgres(1)
Main PID: 6688 (postgres)
CGroup: /system.slice/postgresql.service
├─6688 /opt/data/pgsql/10.15/bin/postgres -D /opt/data/postgres/data/10/data
├─6689 postgres: logger process
├─6696 postgres: checkpointer process
cat /etc/systemd/system/postgresql.service
[Unit]
Description=PostgreSQL database server
Documentation=man:postgres(1)
[Service]
Type=notify
User=postgres
ExecStart=/opt/data/pgsql/10.15/bin/postgres -D /opt/data/postgres/data/10/data
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGINT
TimeoutSec=0
[Install]
WantedBy=multi-user.target
So please can any one help me to get desired output as Active: active (running))
I tried comment the Type=Notify line and it works fine.
Only for PostgreSQL 9x.
Version 10 and above I've no problem.

postgresql could not properly start exited with active (exited)

I installed posgresql from digitalocean and in the end of installation prints the below command in terminal
/usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start
I tried to run it with sudo root user and also with switching to postgres user but gives me below error
waiting for server to start..../bin/sh: 1: cannot create logfile:
Permission denied stopped waiting pg_ctl: could not start server
but when i check the status it says
● postgresql.service - PostgreSQL RDBMS Loaded: loaded
(/lib/systemd/system/postgresql.service; enabled; vendor preset:
enabled) Active: active (exited) since Thu 2018-05-31 13:11:18 UTC;
56s ago Main PID: 3698 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 2362) CGroup: /system.slice/postgresql.service
May 31 13:11:18 staging systemd1: Starting PostgreSQL RDBMS... May
31 13:11:18 staging systemd1: Started PostgreSQL RDBMS.
Status is not running except is exited.what the above command do and how can i run it ? I haven't faced it in previous versions
The idea is that you supply your actual log file instead of logfile, but I recommend that you configure logging properly in postgresql.conf and use pg_ctl without the -l option.
Set logging_collector to on.
Set log_filename to postgresql-%a.log.
Set log_rotation_size to 0.
Set log_truncate_on_rotation to on.
Then you'll get the log files in the log subdirectory of your PostgreSQL data directory, and they will be rotated on a weekly basis.

Cant access Postgres, getting an error about “/var/run/postgresql/.s.PGSQL.5432”

I was running a Django application with Postgres as backend database. It was working fine. All of a sudden, today I saw my database connection is refusing in the production server. So I logged into my server and tried to:
psql
and then it was showing this error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
The file listed above does not seem to exist.
I checked if my Postgres is running or not with:
/etc/init.d/postgresql status
and it was returning with SUCCESS message.
postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2017-11-16 10:03:41 UTC; 55min ago
Process: 4701 ExecReload=/bin/true (code=exited, status=0/SUCCESS)
Process: 4899 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 4899 (code=exited, status=0/SUCCESS)
Nov 16 10:03:41 median systemd[1]: Starting PostgreSQL RDBMS...
Nov 16 10:03:41 median systemd[1]: Started PostgreSQL RDBMS.
Still, I stopped and started postgres again.
I checked if there any other postgres task running or not with:
ps -ef | grep postgres
And it returned:
root 5210 5117 0 10:21 pts/0 00:00:00 grep --color=auto postgres
So there is no other Postgres.
I even checked if my Postgres accidentally got deleted or not with
dpkg -l | grep postgres
and it returned
ii postgresql 9.5+173ubuntu0.1 all object-relational SQL database (supported version)
ii postgresql-9.5 9.5.10-0ubuntu0.16.04 amd64 object-relational SQL database, version 9.5 server
ii postgresql-client-9.5 9.5.10-0ubuntu0.16.04 amd64 front-end programs for PostgreSQL 9.5
ii postgresql-client-common 173ubuntu0.1 all manager for multiple PostgreSQL client versions
ii postgresql-common 173ubuntu0.1 all PostgreSQL database-cluster manager
ii postgresql-contrib-9.5 9.5.10-0ubuntu0.16.04 amd64 additional facilities for PostgreSQL
So there is no rc instead of ii, which means it has not been uninstalled.
I have tried every other solution in the internet but uninstalling Postgres. Didn't work for me. I might be doing something very silly. I don't want to uninstall and lose my data.
Run the following command to check which port your PostgreSQL server is actually running on:
[root#server ~]# netstat -a | grep .s.PGSQL
unix 2 [ ACC ] STREAM LISTENING 2534417 /var/run/postgresql/.s.PGSQL.5432
I can see from this output that my server is listening on port 5432 and also that the file is created in /var/run/postgresql/.s.PGSQL.5432
You can then update /etc/init.d/postgresql to include the lines
PGPORT=5432
export PGPORT
and update your postgresql.conf file which is located in your data directory and set the port key/value to port=5432
and then restart your PostgreSQL server
[root#server] service postgresql restart

Fatal error starting postgres

I'm unfamiliar with how to use postgres and need some help. I'm currently running OSX Yosemite.
When I start postgres I get this:
pg_ctl: could not start server
Examine the log output.
There was an error executing [start] on postgres. Check /Users/work/git/proj/var/log/postgres.log for details.
createuser: could not connect to database postgres: FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory
The log is below.
When I try to stop postgres I get this:
Postgres not running
And when I run ps -ef |grep postgres I get this:
20010 13398 1 0 Jul07 ? 00:00:00 /usr/pgsql-9.3/bin/postgres -h -k /Users/work/git/proj/var/pg
20010 13399 13398 0 Jul07 ? 00:00:09 postgres: logger process
20010 13401 13398 0 Jul07 ? 00:00:10 postgres: checkpointer process
20010 13402 13398 0 Jul07 ? 00:00:00 postgres: writer process
20010 13403 13398 0 Jul07 ? 00:00:00 postgres: wal writer process
20010 13404 13398 0 Jul07 ? 00:00:36 postgres: autovacuum launcher process
20010 13405 13398 0 Jul07 ? 00:00:02 postgres: stats collector process
20010 18112 17723 0 10:22 pts/0 00:00:00 grep postgres
What does this all mean and how could I possibly fix this?
log text
Postgres data dir doesn't exist. Creating
The files belonging to this database system will be owned by user "rose.smith".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /Users/work/git/proj/postgres ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
creating configuration files ... ok
creating template1 database in /Users/work/git/proj/postgres/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/pgsql-9.3/bin/postgres -D /Users/work/git/proj/postgres
or
/usr/pgsql-9.3/bin/pg_ctl -D /Users/work/git/proj/postgres -l logfile start
waiting for server to start....< 2015-06-04 17:24:57.966 GMT >LOG: redirecting log output to logging collector process
< 2015-06-04 17:24:57.966 GMT >HINT: Future log output will appear in directory "pg_log".
done
server started
waiting for server to shut down.... done
server stopped
waiting for server to start....< 2015-06-04 18:10:18.044 GMT >LOG: redirecting log output to logging collector process
< 2015-06-04 18:10:18.044 GMT >HINT: Future log output will appear in directory "pg_log".
done
server started
"/Users/work/git/proj/var/log/postgres.log" 413L, 20935C
after running /usr/pgsql-9.3/bin/postgres -D /Users/work/git/proj/postgres
< 2015-07-08 14:40:36.331 GMT >FATAL: lock file "postmaster.pid" already exists
< 2015-07-08 14:40:36.331 GMT >HINT: Is another postmaster (PID 18145) running in data directory "/Users/work/git/proj/postgres"?
I can't speak to why this worked after trying these commands just a few minutes ago, but it is now working. Good luck to anyone else with the same problem.
stop postgres
killall postgres
remove postgres database with rm -rf postgres
start postgres
This website was helpful. I think my problem may have been the same as his.
I had deleted ~/Library/Containers/com.heroku.postgres or ~/Application Support/Postgres/ while the Postgres.app was still running. The old version was still running since I deleted the pid file, and it didn't know how to shut it down.
Source: https://github.com/PostgresApp/PostgresApp/issues/96
I faced same issue. I solved the problem with the following commands.
If you install postgresql using HomeBrew...
rm /usr/local/var/postgres/postmaster.pid
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
Hope this helps you!