Percona XtraDB Cluster : Server quit without updating PID file ... server startup failed - centos

Can some one help me please ?
1) Host of my VPS (Centos) was down (Provider problem)
2) After VPS reboot, the node XtraDB cluster refuse to start
3) Other nodes (2) were note impacted
I read lot of posts and try to by many ways but nothing works.
i tried simple start
i tried to remove the /var/lib/mysql/xxxxxx.com.pid and try to start
i tied to remove the lock file
always the same result :
#/etc/init.d/mysql start
Starting MySQL (Percona XtraDB Cluster).. ERROR! The server quit without updating PID file (/var/lib/mysql/xxxxxx.com.pid).
ERROR! MySQL (Percona XtraDB Cluster) server startup failed!
or
#/etc/init.d/mysql start
ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster).....State transfer in progress, setting sleep higher
.... ERROR! The server quit without updating PID file (/var/lib/mysql/xxxxxxx.com.pid).
ERROR! MySQL (Percona XtraDB Cluster) server startup failed!
or
#/etc/init.d/mysql start --wsrep-cluster-address="gcomm://"
ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster).. ERROR! The server quit without updating PID file (/var/lib/mysql/xxxxx.com.pid).
ERROR! MySQL (Percona XtraDB Cluster) server startup failed!

I had the same error and none of the other solutions worked for me. However I had a feeling that it had something to do with running out of disk space.
What worked for me was removing the ibdata1, ib_logfile0, and ib_logfile1 in the /var/lib/mysql directory.
The files were re-created on startup. the two logfiles were exactly the same size as previous for the ibdata1 file was half the size.
Someone may be able to confirm, but I believe the ibdata1 file met/exceeded the configured size (I did not set up this instance).

Related

Broken configuration in Mongo on ubuntu - cannot start mongod with correct config

I have managed to break what was a stable instance of Mongo running on an Ubuntu server.
It doesn't seem to start the service using the correct config.
Running mongod gives me the following:
2016-11-01T16:06:27.853+0000 I STORAGE [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating
and therefore, when I try to run the mongo shell I get:
2016-11-01T16:06:48.476+0000 W NETWORK Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
The config file at /etc/mongod.conf states the dbpath as /var/lib/mongod which has a bunch of dbses already in there (which were working!) and the port as 3306. This is clearly not what the error messages above are pointing to.
I've tried running mongod with this config file using --config /etc/mongod.conf but I get this:
2016-11-01T16:09:12.530+0000 F CONTROL Failed global initialization: FileNotOpen Failed to open "/var/log/mongodb/mongod.log"
Any ideas of what steps I can take to restore the original service on the right dbpath and port?
There is an upstart file at /etc/init/mongod.conf but a server reboot hasn't had any impact.
Thanks.
When running mongod with data files on a low disk space volume (typically I had this issue on my always crowded development laptop), you may experience the above, with the slightly misleading symptoms. From memory, the required space in my/our case was around a few gigabytes, 3 typically sufficed.
If this fits, you can either choose to delete files, or to ignore (since you aren't in a production environment), use the --smallfiles option, see the documentation.
(I just noticed that this issue is about 2 years old ... possibly only relates to the mmapv1 engine, which isn't the default since 3.2. Having written it, I'll post this possible answer anyway, probably won't be of much use though by now :) )

Postgresql postmaster takes very long time to start

My situation in Postgres 9.1 is that
I had some running queries which take a lot of memory, then I restarted Postgres immediately. After that I used postmaster to start it again.
/usr/pgsql-9.1/bin/postmaster -p 5432 -D /var/lib/pgsql/9.1/data
However, it got stuck for a very long time without starting successfully.
From Postgres documentation, when we restart server immediately, it may prevent postmaster from freeing the system resources (e.g., shared memory and semaphores).
So what should I do now? or just wait for postmaster starting?
Much depends on how you stopped the server.
If you kill -9 the postmaster process, all the spawned processes (1 for each session) that are using the resources keep on running so your system is still overloaded. Furthermore, the postmaster process writes its pid to a file very aptly called postmaster.pid in the data directory of the cluster (/var/lib/pgsql/9.1/data in your case). When a file with that name is found, a new postmaster process can not start, so you should delete that file. All-in-all, NEVER kill the postmaster.
If you used pg_ctl or the rc.d script to restart the server then you have some other issue on your server. Reboot the server if you can.
In either case, you should not start the server by running the postmaster manually. Use the rc.d script or pg_ctl instead.
Lastly, 9.1 is getting old. Consider upgrading to 9.5 which has lots more features.

What are the practical differences between mongo and mongod?

Just finished installing mongodb, however, I have not been able to make complete sense of the difference between mongo vs mongod commands. Yes, I do understand that
mongod is the primary daemon process for the MongoDB system
and that
mongo is an interactive JavaScript shell interface to MongoDB
but what does that mean practically? I presume every time I want to use mongodb, I need to run mongod first. But then why am I able to run mongo without having started mongod first? Does mongo run mongod in the background automatically? Secondly, if I run mongod it eventually ends with something like
waiting for connections on port 27017
but then I can't type anything after that. Again, I presume that mongodb has been started in the background so I can safely close the terminal. But if I close the terminal by mistake (on a mac), how can I get that back up on the terminal? Also, how can I terminate the service for it to stop listening to the port?
So as you can I see, I have a bunch of simple questions... but most are related to the practical uses of when and when not to mongo or mongod. I can't seem to find anything online that will help me explain these in the practical sense.
As with most database software, Mongo is split into a server and client. The server is the main database component which stores and manages data. Clients come in various flavours and connect to the server to insert or query data.
mongod is the server portion. You start it, it runs, end of story.
mongo is a default command line client. You start it, you connect to a server, you enter commands, you quit it.
You have to run mongod first, otherwise you have no database to interact with. Simply running mongod on a command line will make it the frontmost running application, and it does not offer any interactivity. So yes, you'll just see something like "Waiting for connections...", and nothing more. You typically don't run mongod like that on the command line. You most typically create an init.d script or launchd file or however you manage your daemons, and have the system start it automatically at system boot time.
If you want to launch mongod as a one-off thing without having it permanently running on your systems, put it in the background:
$ mongod &
The & puts it in the background and you can continue to use your command line. You can see it and kill it like this:
~ deceze$ mongod &
[1] 1065
~ deceze$ jobs
[1]+ Running mongod &
~ deceze$ kill %1
[1]+ Done mongod
Once your server is running, start mongo, connect to the server, and interact with it. If you try to run mongo without a running server, it should complain that it's not able to connect:
~ deceze$ mongo
MongoDB shell version: 3.0.2
connecting to: test
2015-08-13T09:36:13.518+0200 W NETWORK Failed to connect to 127.0.0.1:27017, reason: errno:61 Connection refused
2015-08-13T09:36:13.521+0200 E QUERY Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed
at connect (src/mongo/shell/mongo.js:179:14)
at (connect):1:6 at src/mongo/shell/mongo.js:179
exception: connect failed
If your mongo shell does connect to something, you might unknowingly have another instance of mongod running on your system.
With mongodyou are starting the server on your machine. As you stated correctly, mongo is your client, your user interface, if you want to. Per default it connects to your local instance of MongoDB. If you start your client without a server instance running, you would have to 'tell' it, where it should connect to (e.g. a remote instance):
http://docs.mongodb.org/manual/reference/program/mongo/

could not connect to server with postgresql is the server running locally and PGSQL.5432

I´ve read many similar forum discussions about this but I have not been able to solve the problem. I installed postgresql 9.1 on ubuntu 12.04 (not really a server machine, but virtualized in VirtualBox). Yesterday I was able to restart (and start) postgresql server but not today after many trials.
In postgresql.conf i have listen_addresses='*' and port=5432 and unix_socket_directory = 'var/run/postgresql'
I´ve looked into that directory but it seems to be empty.
I´ve tried: locate PGSQL.5432 but there´s no results
My server ip is set like 192.168.1.211 and ping -c 4 192.168.1.211 seems to be ok.
Tried many solutions (like chmod 777 /var/run/postgresql) and nothing seems to fix it.
It´s maybe and stupid issue and I´m not being able to solve it as i don´t have a high level of knowledge on those systems.
Full error messages: after sudo service postgresql restart -- > The postgresql server failed to start. Please check the log output. After sudo psql -- > could not connect to server: the file or directory does´nt exist. Is the server running locally and accepting connections on Unix domain socke "var/run/postgresql/.s.PGSQL.5432?
Postgresql version: 9.1
Just after starting ubuntu, this is the log that i get with tail /var/log/postgresql/postgresql-9.1-main.log:
CET LOG: could not open temporary statistics file "pg_stat_tmp/pgstat.tmp": permission denied.
CET LOG: received smart shutdown request
CET LOG: autovacuum launcher shutting down.
CET LOG: shutting down
CET PANIC: could not open control file "global/pg_control": permission denied
CET LOG: background writer process (PID 7151) was terminated by signal 6: Aborted
CET LOG: terminating any other active server processes
CET LOG: could not open temporary statistics file "pg_stat_tmp/pgstat.tmp": permission denied.
CET LOG: abnormal database systema shutdown.
Any idea about what to try?
Here the solution I´ve found, hope could help other users with similar issue:
I´ve just made:
sudo chown -R postgres:postgres /var/lib/postgresql
And now postgresql is starting fine. So wrong permissions, maybe modified by myself, I´m not sure.
Kind regards.
I faced the same issue, this was solved after making the changes in the postgresql.conf i.e.
listen_addresses='*'
And restarted the device it was solved,for more detail use the below given link, Restart the device after making changes in the config file it solves most of the firewall issues.
Help
This error generally means that the server is not running. Based on dpkg -l output and the thread of comments, it was due to the postgresql-9.5 main package being somehow uninstalled. Since the uninstall hasn't been called with the --purge option to dpkg, the data and configuration files are still there, so apt-get install postgresql-9.5 can fix the problem.

postgres sometimes will not start after reboot

Something weird is happening with my postgres installation after I upgraded to version 9.3.2 homebrew.
Sometimes and not every time, if I enter psql I get this error message:
could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
It also appears that postgres is running because if I try and stop postgres with this command:
pg_ctl -D /usr/local/var/postgres stop -s -m fast
I get this response:
pg_ctl: server does not shut down
If I look in the server.log I can see the following entries:
FATAL: lock file "postmaster.pid" already exists HINT: Is another
postmaster (PID 208) running in data directory
"/usr/local/var/postgres"?
After some frantic googling, I am able to cure this by entering these commands:
launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
rm ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
I would like to first of all understand what is happening and second of all I would like to fix it once and for all.
Can anyone explain what might be happening and a cure?
First, try to pull up a log on PostgreSQL, on OSX (and with PG 9.3) this will most likely be here:
/Library/PostgreSQL/9.3/data/pg_log
Check this log and see if there are any entires in it that may explain more of what is happening. Post it here or in a past bin somewhere. This can help the community to debug the issue.
Next, the reason why 'pg_ctl' says the server is not shutdown is because you have a stale socket file and/or a stale .plist file that did not get removed on last shutdown. This may indicate that the PostgreSQl daemon was "crashed" or was shutdown by force and did not have a clean finish.
The commands you used removed the stale files and thus made way for a new socket file to be created.
Also it is important to know if this was a minor or a major upgrade of PostgreSQL. Major upgrades always require a migration of the data directory. Running an old PostgreSQL data directory with a new engine might have unexpected results.