Postgres server running on EC2 closing every once in while. Unable to isolate the issue here - postgresql

I have a database connection configured on my Django application to a Postgres server.
This server is running on an EC2 instance.
Every once in a while (3 times as of this post) I see this when I run my django App
Is the server running on host "xx.com" (13.xx.xx.xx) and accepting
TCP/IP connections on port 5432?
When I run this : sudo service postgresql status
I get
postmaster dead but pid file exists
and when I run sudo service postgresql restart I get
Stopping postgresql service: [FAILED]
Starting postgresql service: [ OK ]
I tried deleting postmaster.pid file in /var/lib/pgsql9/data
But that does not seem to solve the problem.
It seems to work fine when I restart the server. But that is Windows-user type solution I am using.
-- I am still new to using postgres.
Here is what the log file (in /var/lib/pgsql9/data/pg_log) had when the server went down:
sh: line 46: kill: (5102) - No such process
sh: line 46: kill: (5103) - No such process
sh: line 46: kill: (5105) - No such process
sh: line 46: kill: (5106) - No such process
sh: line 46: kill: (5126) - No such process
sh: line 46: kill: (5127) - No such process
sh: line 46: kill: (5130) - No such process
sh: line 46: kill: (5150) - No such process
sh: line 46: kill: (5151) - No such process
sh: line 46: kill: (5154) - No such process

Your application is clearly not able to reach to the server.
The error states that the application/client is trying to find a server on xx.com where it cannot find the postmaster service running on port 5432 and accepting connections.
From the other information you provided, it looks like there's something killing the postmaster at backend and when this happens the postmaster.pid file is not removed. And as long as there's a postmaster.pid file, the service status will show this message postmaster dead but pid file exists.
Try looking into the database logs (/pg_log) for more information on the crash part.

Related

pymongo ServerSelectionTimeoutError

I'm doing some tutorial about docker-compose with flask and mongodb. When I run command "docker-compose up", everything is fine. But when I send a post request to insert a json data to database I have Timeout Error like this:
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s
I guess that I haven't started database yet. But when I try to put command "sudo systemctl stop mongod" in to docker-compose.yml file, I got an error:
db_1 | /usr/local/bin/docker-entrypoint.sh: line 363: exec: sudo: not found
Does anybody know how to fix it?

Mongodb 3.4 - fail to start mongodb

How can I start mongodb. The child process fail.
-bash-4.1$ mongod --config /mongodb/etc/mongod-ctr.conf
about to fork child process, waiting until server is ready for connections.
forked process: 22965
ERROR: child process failed, exited with error number 48
-bash-4.1$ mongo
MongoDB shell version v3.4.17
connecting to: mongodb://127.0.0.1:27017
2018-11-27T13:52:13.826-0500 W NETWORK [thread1] Failed to
connect to 127.0.0.1:27017, in(checking socket for error after
poll), reason: Connection refused
According to the MongoDB error code manual, and your error status, there seems to be another process either running on port 27017, or you just don't have the right permissions.
You can use the command lsof -i -P -n to see a list of all applications, and the ports they are using.

PostgreSQL - Fail to start delegate ip address on pgpool 2

I try to set up a pgpool server on ubuntu server and following this link : pgpool-II Tutorial [ Watchdog ].
But when I to start a pgpool service, the delegated IP doesn't start.
I have seen in a log file on syslog and got some error like this.
Oct 25 08:46:25 pgpool-1 pgpool[1647]: [8-2] 2017-10-25 08:46:25: pid 1647: DETAIL: Host:"172.16.0.42" WD Port:9000 pgpool-II port:5432
Oct 25 08:46:25 pgpool-1 pgpool: SIOCSIFADDR: Operation not permitted
Oct 25 08:46:25 pgpool-1 pgpool: SIOCSIFFLAGS: Operation not permitted
Oct 25 08:46:25 pgpool-1 pgpool: SIOCSIFNETMASK: Operation not permitted
Oct 25 08:46:25 pgpool-1 pgpool[1648]: [18-1] 2017-10-25 08:46:25: pid 1648: LOG: failed to acquire the delegate IP address
Oct 25 08:46:25 pgpool-1 pgpool[1648]: [18-2] 2017-10-25 08:46:25: pid 1648: DETAIL: 'if_up_cmd' failed
Oct 25 08:46:25 pgpool-1 pgpool[1648]: [19-1] 2017-10-25 08:46:25: pid 1648: WARNING: watchdog escalation failed to acquire delegate IP
I use ubuntu 14.04 with pgpool2 version 3.6.6-1, and watchdog version 5.31-1.
And I has configured on pgpool.conf at virtual IP setting like this.
# - Virtual IP control Setting -
delegate_IP = '172.16.0.201'
if_cmd_path = '/sbin'
if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.0.0'
if_down_cmd = 'ifconfig eth0:0 down'
arping_path = '/usr/sbin'
arping_cmd = 'arping -U $_IP_$ -w 1'
Any suggestion for this? Thank you for any help.
Looks like user that runs it doesn't have permission to use ifconfig.
Did you follow those steps from tutorial?
setuid configuration
In watchdog process, root privilege is required to contol virtual IP.
You could start pgpool-II as root user. However in this tutorial,
Apache needs to start pgpool as apache user and control virtual IP
because we are using pgpoolAdmin. For this purpose, we setuid
if_config and arping. Also we don't want any user other than apache
accesses the commands because of security reason. Execute following
commands on each of osspc19 and osspc20 (It requires root privilege).
At first, make a directory for containing ipconfig and arping which is
set setuid. The path is specified at ifconif_path and arping_path; in
this tutorial, this is /home/apache/sbin. Then give execute privilege
to only apache user.
$ su -
# mkdir -p /home/apache/sbin
# chown apache:apache /home/apache/sbin
# chmod 700 /home/apache/sbin
Next, copy the original ifconfig and arping to the directory and then
set setuid to these.
# cp /sbin/ifconfig /home/apache/sbin
# cp /use/sbin/arping /home/apache/sbin
# chmod 4755 /home/apache/sbin/ifconfig
# chmod 4755 /home/apache/sbin/arping
Note that explained above should be used for tutorial purpose only. In
the real world you'd better create setuid wrapper programs to execute
ifconfig and arping. This is left for your exercise.
(Note: this answer may help in case you run Pgpool-II servers with Watchdog in Docker containers)
I tried to setup Pgpool-II servers with Watchdog in Docker containers today, and I got almost the same error (though I did set the SUID bit and even tried running Pgpool-II as the root user):
SIOCSIFADDR: Operation not permitted
SIOCSIFFLAGS: Operation not permitted
SIOCSIFNETMASK: Operation not permitted
pid 88: LOG: failed to acquire the delegate IP address
pid 88: DETAIL: 'if_up_cmd' failed
pid 88: WARNING: watchdog escalation failed to acquire delegate IP
Later I found that it was because the container did not have the privilege to change its network configurations, by default by design.
I then ran my Pgpool-II Docker containers in the privileged mode as shown below:
pgpool1:
privileged: true
image: postdock/pgpool:latest-pgpool36
...
The error is gone and the virtual IP is set up correctly.
My problem is solved by the following method.
Make a directory for containing ipconfig and arping. Then give execute privilege to only non-root user.
$mkdir /var/lib/pgsql/sbin
$chown postgres:postgres /var/lib/pgsql/sbin
$cp /sbin/ip /var/lib/pgsql/sbin
$cp /sbin/arping /var/lib/pgsql/sbin
Run visudo, which safely edits the sudoers file:
$visudo
Then add an entry like this in sudoers file:
postgres ALL = NOPASSWD: /var/lib/pgsql/sbin/ip *, /var/lib/pgsql/sbin/arping *
Next, create bash files(ipadd.sh,ipdel.sh,arping.sh) to run ip and arping commands with sudo.
$cat /var/lib/pgsql/sbin/ipadd.sh
#!/bin/bash
sudo /var/lib/pgsql/sbin/ip addr add $1/24 dev eth1 label eth1:0
$cat /var/lib/pgsql/sbin/ipdel.sh
#!/bin/bash
sudo /var/lib/pgsql/sbin/ip addr del $1/24 dev eth1
$cat /var/lib/pgsql/sbin/arping.sh
#!/bin/bash
sudo /var/lib/pgsql/sbin/arping -U $1 -w 1 -I eth1
$chmod 755 /var/lib/pgsql/sbin/*
$chown postgres:postgres /var/lib/pgsql/sbin/*
Add an entry like this in pgpool.conf:
delegate_IP = '10.10.10.62'
if_up_cmd = 'ipadd.sh $_IP_$'
if_down_cmd = 'ipdel.sh $_IP_$'
arping_cmd = 'arping.sh $_IP_$'
if_cmd_path = '/var/lib/pgsql/sbin'
arping_path = '/var/lib/pgsql/sbin'
Then restart the pgpool service. Ignore the warning you can see as follows.
WARNING: checking setuid bit of if_up_cmd
DETAIL: ifup[/var/lib/pgsql/sbin/ipadd.sh] doesn't have setuid bit
WARNING: checking setuid bit of if_down_cmd
DETAIL: ifdown[/var/lib/pgsql/sbin/ipdel.sh] doesn't have setuid bit
WARNING: checking setuid bit of arping command
DETAIL: arping[/var/lib/pgsql/sbin/arping.sh] doesn't have setuid bit
Stop and check one of your two pgpool services.

iRODS configuration - Could not start iRODS server during

I've installed postgres as database and then iRODS in Ubuntu 14.04. Then I start its configuration
sudo /var/lib/irods/packaging/setup_irods.sh
After the configuration phase, when iRODS starts the updtating, the first 4 steps go well
Stopping iRODS server...
-----------------------------
Running irods_setup.pl...
Step 1 of 4: Configuring database user...
Updating user's .pgpass...
Skipped. File already uptodate.
Step 2 of 4: Creating database and tables...
Checking whether iCAT database exists...
[mydb] on [localhost] found.
Updating user's .odbc.ini...
Creating iCAT tables...
Skipped. Tables already created.
Testing database communications...
Step 3 of 4: Configuring iRODS server...
Updating /etc/irods/server_config.json...
Updating /etc/irods/database_config.json...
Step 4 of 4: Configuring iRODS user and starting server...
Updating iRODS user's ~/.irods/irods_environment.json...
Starting iRODS server...
but at the end I get this error
Could not start iRODS server.
Starting iRODS server...
Traceback (most recent call last):
File "/var/lib/irods/iRODS/scripts/python/get_db_schema_version.py", line 77, in <module>
current_schema_version = get_current_schema_version(cfg)
File "/var/lib/irods/iRODS/scripts/python/get_db_schema_version.py", line 61, in get_current_schema_version
'get_current_schema_version: failed to find result line for schema_version\n\n{}'.format(format_cmd_result(result)))
RuntimeError: get_current_schema_version: failed to find result line for schema_version
return code: [0]
stdout:
stderr:
ERROR: relation "r_grid_configuration" does not exist
LINE 1: ...option_value from R_GRID_CON...
^
Confirming catalog_schema_version... Success
Validating [/var/lib/irods/.irods/irods_environment.json]... Success
Validating [/etc/irods/server_config.json]... Success
Validating [/etc/irods/hosts_config.json]... Success
Validating [/etc/irods/host_access_control_config.json]... Success
Validating [/etc/irods/database_config.json]... Success
(1) Waiting for process bound to port 5432 ... [-]
(2) Waiting for process bound to port 5432 ... [-]
(4) Waiting for process bound to port 5432 ... [-]
Port 5432 In Use ... Not Starting iRODS Server
Install problem:
Cannot start iRODS server.
Found 0 processes:
There are no iRODS servers running.
Abort.
Have you any ideas on what went wrong?
Because I don't have enough reputation to comment:
Which version of iRODS are you using?
This portion of the output:
Creating iCAT tables...
Skipped. Tables already created.
combined with this portion:
ERROR: relation "r_grid_configuration" does not exist
suggests that the setup ran before, but only partially completed, leaving the system in a broken state. I would recommend reinstallating from scratch, which includes:
Uninstalling the iRODS icat and db plugin packages:
sudo dpkg -P irods-icat irods-database-plugin-postgres
note: make sure to use the -P, so that the configuration files are removed from dpkg's database.
Dropping and remaking the database
Deleting the following directories:
sudo rm -rf /tmp/irods /etc/irods /var/lib/irods
Reinstalling the packages and running sudo /var/lib/irods/packaging/setup_irods.sh
This portion of the output:
(1) Waiting for process bound to port 5432 ... [-]
(2) Waiting for process bound to port 5432 ... [-]
(4) Waiting for process bound to port 5432 ... [-]
Port 5432 In Use ... Not Starting iRODS Server
suggests that you are using port 5432 as your iRODS server port. This will conflict with the default Postgres port. I recommend using the default iRODS server port of 1247. This value was queried during setup as:
iRODS server's port [1247]:
and is recorded in /etc/irods/server_config.json under the zone_port entry.
iRODS-Chat:
It may be easier to continue this on the iRODS-Chat google group. Repairing installs can require back-and-forth communication, which may not be inline with standard stackoverflow usage.

Shutdown mongos for upgrade

I am upgrading a mongo sharded cluster, and am in the first step of replacing the mongos process. Can I just kill this process or is there a safer way to shut it down before replacing it?
It is the answer. But may be you misunderstood. You cannot shutdown mongos from command line, but you have to logon to the shell. Here is the result I am trying on my own:
$ mongo --port 27077 (please replace with your own port on mongos instance)
**mongos**> db.shutdownServer({timeoutSecs:30})
shutdown command only works with the admin database; try 'use admin'
mongos> use admin
switched to db admin
mongos> db.shutdownServer({timeoutSecs:30})
2015-02-27T12:53:54.408+0800 DBClientCursor::init call() failed
**server should be down...**
2015-02-27T12:53:54.410+0800 trying reconnect to 127.0.0.1:27077 (127.0.0.1) failed
2015-02-27T12:53:54.410+0800 warning: Failed to connect to 127.0.0.1:27077, reason: errno:111 Connection refused
2015-02-27T12:53:54.410+0800 reconnect 127.0.0.1:27077 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27077 (127.0.0.1), connection attempt failed
2015-02-27T12:53:54.413+0800 trying reconnect to 127.0.0.1:27077 (127.0.0.1) failed
2015-02-27T12:53:54.413+0800 warning: Failed to connect to 127.0.0.1:27077, reason: errno:111 Connection refused
2015-02-27T12:53:54.413+0800 reconnect 127.0.0.1:27077 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27077 (127.0.0.1), connection attempt failed
>
Try to run "top" or "htop" to display all processes running on your computer. You shall see the "mongos" process has gone. Read the log file. Here is the last two lines of my log:
2015-02-27T12:53:54.406+0800 [conn1] terminating, shutdown command received
2015-02-27T12:53:54.406+0800 [conn1] dbexit: shutdown called rc:0 shutdown called
And here is the link of my testing script:
https://github.com/babycaseny/QuickStart/blob/master/StartShard.sh
Note that you have to replace the "localhost" in the mongod/mongos command with the hostname of your computer, or you will not be able to config your shards.
See also this one:
https://groups.google.com/forum/#!topic/mongodb-user/TQLlRI6HG1M
In case you need a command line to do the work, here is one:
mongo admin --port portnumber --eval "db.shutdownServer()"
Notice that you have to run the command in localhost.