Gitlab not starting after upgrade to Ubuntu 18.04 - postgresql

I have successfully upgraded Gitlab to 12.1.6 on Ubuntu 16.04 and checked that all was working. After making sure Ubuntu was fully up to date I checked again: Gitlab worked.
I then used the do-release-upgrade command to update to Ubuntu 18.04. After restart, everything seems to work ok, but Gitlab refuses to start.
I get the following errors:
fail: alertmanager: runsv not running
fail: gitaly: runsv not running
fail: gitlab-exporter: runsv not running
fail: gitlab-workhorse: runsv not running
fail: grafana: runsv not running
fail: logrotate: runsv not running
fail: nginx: runsv not running
fail: node-exporter: runsv not running
fail: postgres-exporter: runsv not running
fail: postgresql: runsv not running
fail: prometheus: runsv not running
fail: redis: runsv not running
fail: redis-exporter: runsv not running
fail: sidekiq: runsv not running
fail: unicorn: runsv not running
I tried:
gitlab-ctl reconfigure --> runs with success
I installed runit with success, rebooted the machine but the errors remain
I found a similar issue here: on Stackoverflow, followed the instructions (yum --> apt), still no success
and here on Gitlab. This advised to run
sudo systemctl restart gitlab-runsvdir
sudo gitlab-ctl restart
But the first command never finishes
I found this: on Gitlab which states to run
sudo gitlab-rake gitlab:env:info --trace
Output:
** Invoke gitlab:env:info (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:env:info
System information
System: Ubuntu 18.04
Current User: git
Using RVM: no
Ruby Version: 2.6.3p62
Gem Version: 2.7.9
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 3.2.12
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.1.4/lib/pg.rb:56:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.1.4/lib/pg.rb:56:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.1.4/lib/pg.rb:56:in `connect'
which suggests that the Postgresql server is not running. I have no idea how to start it. Any ideas?

UPDATE: Since then I found an answer that said for some reason on the reboot command the runsv process gets stuck. To solve this you have to:
sudo service gitlab-runsvdir start
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
I wanted to post the same above question. This happens to me when i do the monthly upgrade/updates to our GitLab. I tried your solution before, but to me they didn't solve the problem.
I present my solution, but use these tips with caution, as I still sometimes struggle with this problem in different ways.
I do a combination of these command, the order of these steps are really important(!):
sudo systemctl disable gitlab-runsvdir.service
sudo gitlab-ctl reconfigure
sudo gitlab-rake db:migrate
sudo gitlab-ctl restart gitlab-runsvdir--has to be aborted with Ctrl-C
sudo /opt/gitlab/embedded/bin/runsvdir-start--has to be aborted with Ctrl-C
Reference:
In this issue Stan Hu's answer helped me:
https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3744
Now yesterday I messed up and didn't do the first step, which could cause my problems, because somehow then for 10 hours the postgresql server was down and it couldn't make it back up again.
If i would have stick to the order, maybe it wouldn't have happened, because it didn't happen in the past while I upgraded/updated GitLab several times.
This was the connection error yesterday, same issue as questioner's
Automatic restart this morning
This morning magically the server at 7:13 am, was accepting connections again. (althought i tried to restart it yesterday, it didn't make any difference then)
Still GitLab wasn't reachable online from web browser as of 7:30 am.
One hour later, and a few runsvdir restarting/starting commands after (added it to the commands order), the GitLab is working. I have no idea why.

Got the same problem, and I ended up running /opt/gitlab/embedded/bin/runsvdir-start manually. I fixed my problem, launched reconfigure, and then it worked from there.

I ran into a similar runsv error, but only saw it for once service, not the whole list you have. These steps are a log of my attempts to get it working - probably not a direct line, but my local Gitlab does work now:
In the CentOS vm:
vi /etc/gitlab/gitlab.rb
change the external_url from http://example.gitlab.com to http://192.168.1.131
sudo gitlab-ctl reconfigure
first observed the error runsv not running
yum update -y
sudo gitlab-ctl status
sudo gitlab-ctl restart
sudo gitlab-ctl reconfigure
systemctl start gitlab-runsvdir.service
systemctl status gitlab-runsvdir.service
sudo gitlab-ctl reconfigure
still saw an error about runsv not running, several times, but it was never a blocker and the reconfigure was successful
On host
navigate to 192.168.1.131
See the prompt for root password
As for the issue with Postgres, I'm not sure

Related

5: Input/output error Error: Failure while executing; `/bin/launchctl bootstrap gui/502 and FATAL: password authentication failed for user

The error 5: Input/output Error: Failure while executing; /bin/launchctl bootstrap gui/502 /Users/Andrea/Library/LaunchAgents/homebrew.mxcl.postgresql.plist exited with 5 is occurring when typing in the command "brew services start postgresql".
I am trying to create a database using homebrew and postgresql. If I type the command "brew services stop postgresql" and then start it back up the error does not occur. But will reoccur on the initial start of postgresql.
After, I stop postgresql and restart it using homebrew I type in "createdb 'test'. This error appears:
"error: could not connect to database template1: FATAL: password authentication failed for user "Andrea". Initially I tried to primarily work on the password authentication with no luck.
Here are the steps I have taken:
I have changed the method column to trust in my hba_pg files.
I have tried the command ALTER in the terminal to change the password.
I created a root login on my computer, where I tried to access the file and use the root's password to create the database.
The command chpass to change my user's password.
Help would be much appreciated. Thank you!
This is what I did to resolve on my Mac.
Looked at the postgres.log cat /usr/local/var/log/postgres.log | tail -100
Found the following error on my logs FATAL: lock file "postmaster.pid" already exists
Fix above PID issue and restart the server.
rm /usr/local/var/postgres/postmaster.pid
brew services restart postgresql
Sometimes this can be solved by simply running
brew services restart postgresql
instead of
brew services start postgresql
It happens when a previous postgres process doesn't terminate properly (maybe because you killed a psql process in activity monitor or otherwise). Restarting appears to properly unlock/delete the postmaster.pid file where "start" doesn't.
On the m1 mac I had the same issue and had to remove the postmaster.pid file as well. For m1 macs, the file is at /opt/homebrew/var/postgres/postmaster.pid, so I ran
/opt/homebrew/var/postgres/postmaster.pid
and I could brew services start postgresql successfully.
The other answers didn't apply because I didn't have the pid file (or many other files in those directories). The same error was caused by a bad install, most likely because I had previously installed libpq to get the psql CLI in isolation previously as per: Correct way to install psql without full Postgres on macOS?.
Solution, clean install:
> brew uninstall postgresql
> brew uninstall libpq
> brew install postgresql
> brew services run postgresql
To build on what #gajen-sunthara said, I made sure to upgrade my postgresql installation:
brew services upgrade postgresql
Then removed the postmaster.pid:
rm /usr/local/var/postgres/postmaster.pid
And restarted postgresql:
brew services restart postgresql
And that should get it going again :)
I got the same error:
% brew services start yabai
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/503 /Users/myuser/Library/LaunchAgents/
homebrew.mxcl.yabai.plist` exited with 5.
I have an extra user who manages brew called brew.
After trying to run a brew service with sudo, this was a solution (on Apple silicon):
% sudo brew services stop yabai
% sudo chmod -R g+w /opt/homebrew
% sudo chown -R brew /opt/homebrew
% brew services start yabai
I know this has already been answered, but for anyone seeing this in the future, here is some help.
This often happens because it wasn't properly stopped when it was used at a different time, and the version you are using now has a conflict with what you were running previously. There are a few ways to fix it.
Try stoping it with brew services stop postgresql
Start it with brew services start postgresql
You can also just do brew services restart postgresql and not do step 1 and 2. Hope this helps someone!

brew services. Cant start service. get "Bootstrap failed: 5: Input/output error"

running brew services start mongodb-community which yields:
Bootstrap failed: 5: Input/output error
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/<myUserName>/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist` exited with 5.
when I run launchctl load -w /Users/<myUserName>/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
yields
Load failed: 5: Input/output error
I recently upgraded macOs to BigSur 11.5.2.
Ive uninstall and reinstalled homebrew and xcode.
try this
brew services restart <mongodb-community#4.4>
I am able to run mongodb through homebrew. I believe this solved my issue:
launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
followed by
launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
I saw this issue with MySQL and the solution was my current profile was not the owner of my homebrew folder.
I had to change it because I have a second profile set up for another work space, but they share the same homebrew path.
I would suggest anyone who sees this error try running
sudo chown -R $(whoami) $(brew --prefix)/*
And then seeing if that will resolve it. Again I didn't have this issue with MongoDB, but I saw this exact error with MySQL when my profile didn't own the brew folder.
Try this if you tried to restart your service, but it still doesn't work
stop your service
brew services stop mongodb-community
restart your macOS
start your service again
brew services start mongodb-community
To see the real errors, you need to run the service without brew
I had a similar problem, but instead of mongodb, I could not start apache(httpd).
$ brew services start httpd
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501
/Users/john/Library/LaunchAgents/homebrew.mxcl.httpd.plist` exited with
5.
FAIL: 1
Unfortunately brew doesn't show the user what causes this error. At first I tried to use the tips from the answers in this thread, but neither reinstalling apache nor unload -w/load -w helped.
I also found nothing in the brew logs. I decided to investigate apache itself. To do this I ran apache without brew:
$ /opt/homebrew/Cellar/httpd/2.4.51/bin/apachectl start
httpd: Syntax error on line 66 of /opt/homebrew/etc/httpd/httpd.conf: Cannot load /opt/homebrew/Cellar/php#5.6/5.6.40_3/lib/httpd/modules/libphp5.so into server: dlopen(/opt/homebrew/Cellar/php#5.6/5.6.40_3/lib/httpd/modules/libphp5.so, 0x000A): tried: '/opt/homebrew/Cellar/php#5.6/5.6.40_3/lib/httpd/modules/libphp5.so' (no such file), '/usr/local/lib/libphp5.so' (no such file), '/usr/lib/libphp5.so' (no such file)
FAIL: 1
Then it was finally clear what was the reason why httpd service was not starting. It turned out that after the php upgrade, the php version had changed and I forgot to write the new php version in the apache configuration file.
I quickly fixed this and apache started up.
It seems that the mongodb-community is already started. So do restart it instead of starting, by the following command:
brew services restart mongodb-community
I faced a similar problem with postgresql 11 and restarting the service did not help. The issue arose when my MacBook restarted abruptly.
After checking the ports and trying out all the possible solutions, I decided to look into the logs.
tail -n 100 /usr/local/var/log/postgresql#11.log
This gave me the last 100 lines of the log where I found this:
FATAL: lock file "postmaster.pid" already exists
HINT: Is another postmaster (PID 479) running in data directory "/usr/local/var/postgresql#11"?
Clearly, the issue was with postmaster.pid so I deleted it:
rm /usr/local/var/postgresql#11/postmaster.pid
Started the postgres service again:
brew services start postgresql#11
and voila! It worked.
I had similar error in regard to postgres.
$ brew services start postgresql
Bootstrap failed: 5: Input/output error
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/yrk/Library/LaunchAgents/homebrew.mxcl.postgresql.plist` exited with 5.
I've tried restart and it looked promising:
$ brew services restart postgres
Stopping `postgresql`... (might take a while)
==> Successfully stopped `postgresql` (label: homebrew.mxcl.postgresql)
==> Successfully started `postgresql` (label: homebrew.mxcl.postgresql)
But there still was a problem when trying to connect:
$ psql -d postgres -U apiuser
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
Checking postgres log as mentioned by #mansoor.khan helped, in my case it was:
tail /usr/local/var/log/postgres.log.
And the issue was clear:
$ tail /usr/local/var/log/postgres.log
2022-05-19 14:18:46.029 CEST [32374] FATAL: database files are incompatible with server
2022-05-19 14:18:46.029 CEST [32374] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 14.3.
2022-05-19 14:18:56.071 CEST [32676] FATAL: database files are incompatible with server
2022-05-19 14:18:56.071 CEST [32676] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 14.3.
2022-05-19 14:19:06.111 CEST [32885] FATAL: database files are incompatible with server
2022-05-19 14:19:06.111 CEST [32885] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 14.3.
2022-05-19 14:19:16.128 CEST [33102] FATAL: database files are incompatible with server
2022-05-19 14:19:16.128 CEST [33102] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 14.3.
2022-05-19 14:19:26.166 CEST [33434] FATAL: database files are incompatible with server
2022-05-19 14:19:26.166 CEST [33434] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 14.3.
Now the problem is: how to revert to previous version or upgrade old data to new version of postgres DB, but this is for another topic.
run something like launchctl remove homebrew.mxcl.mongodb-community before the load command solved my problem
I had this when upgrading from 4.4 to 5.0 and I suspect it’s similar for you. Lets assume you were upgrading 5.x to 6.x then the resolution would be ...
check your mongo log, e.g.
vi /usr/local/var/log/mongodb/mongo.log
Look for any errors relating to file/directory permissions and fix those, e.g. though I’m not quite sure whether ‘wheel’ or ‘admin’ is right.
sudo chown yourloginname:wheel thedirectory/*
Roll back to your previous working version of mongo (e.g. 5.x)
Start Mongo 5
Start a Mongo Shell and type
db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )
Check the logs to verify the database has detected 5.0
Uninstall 5.x
Re-install 6.x
Restart / start 6.x
It should now work?
In my case, another httpd process was running on the Mac. Killed with monitor tool allowed me to start httpd.
I had a similar problem on mysql.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/gerd/Library/LaunchAgents/homebrew.mxcl.mysql.plist` exited with 5.
In my case the mysqld miss the osx network connection permission. After restart OSX it appear the network permissions dialog. I granted the permission to msqld and the problem was fixed!
Happened with redis. Realized there was a bind setting in the config, binding the server to an incorrect ip address.
I faced the same problem. This commands helped me
brew uninstall rabbitmq
brew uninstall erlang
brew install rabbitmq
brew services restart rabbitmq
You may run into this if the service is already running, but as another user.
For example, you may have run sudo brew services start example and are now running brew services info example. It tells you it's not running and you think "hm, it should be running" and enter brew services start example.
Potentially example is already bound to, say, port 7777 and now you ask for another program to be bound to the same port, leading to a failing service start.
In short: If you are binding to a port, check if the same service is already running as another user and bound to the same port.

redhat linux sudo su - failed

I am no more able to do sudo su - to get root authentication. I am logged in as test user. I mapped test from unconfined_u to SElinux user staff_u. I restarted the vm machine on google cloud. I can login to vm as test user, but if I do sudo su -, I get following error:
su: avc.c:74: avc_context_to_sid_raw: Assertion `avc_running' failed.
Aborted
It seems to be a known bug of CentOS and Redhat:
0011249: su: avc.c:74: avc_context_to_sid_raw: Assertion `avc_running'
failed.
Description Whenever I execute a su - command after switching
into the sysadm_r, I get this fault. The system is in enforcing mode
when I get this error. When in permissive mode, this does trigger the
error. Attempted to address the problem with audit2allow to see if
that could temporarily resolved the problem while I looked for a
permanent solution but that did not work either
There are the 2 referencies:
Centos bug
RedHat Bug

postgresql#10-main.service failed to load

I have been racking my brain on this issue, and probably have tried every possible solution ( fix, purge, reinstall), but postgresql doesn't start
rupin#linuxbox:~$ sudo /etc/init.d/postgresql restart
[sudo] password for rupin:
Restarting PostgreSQL 10 database server
Failed to issue method call: Unit postgresql#10-main.service failed to load: No such file or directory. See system logs and 'systemctl status postgresql#10-main.service' for details.
The Log file is empty. There is no postmaster.pid file that could be to blame.
My Ubuntu Laptop crashed and the DB was active when the system crashed.
Can someone advise what I can do to fix this issue and start the server again?
**Update **
rupin#linuxbox:~$ systemctl status postgresql#10-main.service
Failed to issue method call: No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/systemd1/unit/postgresql_4010_2dmain_2eservice
Become OS user postgres.
Try to start PostgreSQL with
pg_ctl start -D /etc/postgres/10/main
(Use the path where postgresql.conf resides.)
Then you will see the error and you will know what to do about it.
I followed the instructions of the answer from https://askubuntu.com/questions/873091/postgresql-fails-to-reinstall-after-upgrading-ubuntu-12-04-to-14-04
I ended up removing systemd
sudo apt-get purge systemd && sudo apt-get autoremove
And I was able to reinstall postgres. Unfortunately, I lost my data/cant trace it. It is okay for me, because it is my local DB

Initialise postgres on centos docker image fails with Failed to get D-Bus connection: Operation not permitted

I'm trying to install postgres on a docker image, base image is running centos.
I'm following instructions at How to Install PostgreSQL Relational Databases on CentOS 7 and here's my Dockerfile:
FROM our-internal-docker-registry.org/centos:7.3.1611
RUN yum install -y postgresql-server postgresql-contrib
RUN postgresql-setup initdb
The first RUN command succeeds however the second one fails with:
Step 3/3 : RUN postgresql-setup initdb
---> Running in d26f43467aa0
Failed to get D-Bus connection: Operation not permitted
failed to find PGDATA setting in postgresql.service
The command '/bin/sh -c postgresql-setup initdb' returned a non-zero code: 1
I admit to being clueless as to what to do next as I'm not very familiar with postgres, or indeed linux. Hoping for some advice.
The "D-Bus connection: Operation not permitted" is a hint that it wants to contact the SystemD daemon. I am using postgres on centos along with https://github.com/gdraheim/docker-systemctl-replacement to avoid that.