My company has a legacy, internal server running CentOS 6.4 with postgreSQL 8.4.13. The goal is to upgrade it as far as possible without doing OS updates; it's a live internal server being used for services, so it's not practical to upgrade the OS or have a long downtime. It's old and doesn't follow best practices at all, but unfortunately, this is what I have to work with.
Using the CentOS vault, I can use the base repo to upgrade to 8.4.20.
I can also add and access the pgdg archive repos (for 8.4.22 and up), but they're not included in the CentOS postgresql group and install separately from the default CentOS installation.
I've tried doing that anyway with postgreSQL 8.4.22 as an initial small stop (since "minor" versions < 10 were actually major releases) and pg_upgrade fails with:
The old cluster lacks some required control information:
latest checkpoint oldestXID
If I check pg_controldata, I get
pg_control version number: 843
Catalog version number: 200904091
Database system identifier: 5893982526456722425
Database cluster state: in production
pg_control last modified: Sat 05 Mar 2022 04:35:52 PM JST
Latest checkpoint location: 278A/6517F558
Prior checkpoint location: 278A/6517F510
Latest checkpoint's REDO location: 278A/6517F558
Latest checkpoint's TimeLineID: 1
Latest checkpoint's NextXID: 7/1001247883
Latest checkpoint's NextOID: 260730376
Latest checkpoint's NextMultiXactId: 1
Latest checkpoint's NextMultiOffset: 0
Time of latest checkpoint: Sat 05 Mar 2022 04:35:40 PM JST
Minimum recovery ending location: 0/0
Maximum data alignment: 8
Database block size: 8192
Blocks per segment of large relation: 131072
WAL block size: 8192
Bytes per WAL segment: 16777216
Maximum length of identifiers: 64
Maximum columns in an index: 32
Maximum size of a TOAST chunk: 1996
Date/time type storage: 64-bit integers
Float4 argument passing: by value
Float8 argument passing: by value
There's clearly no reference to latest checkpoint oldestXID in it.
I looked at the changelogs for 8.4.21 and 8.4.22, but there are no references to "oldestXID". I've also tried, using a backup server, pg_resetxlog -f /var/lib/pgsql/data, which yields the same pg_control file without latest checkpoint oldestXID.
I realize that these are all incredibly old versions, but that just means I'm doubly lost here. I hope someone has some ideas, because I'm all out.
So, my friend asked me if I'd gone through the old source code looking for references, and I realized I hadn't because I hadn't been able to find the source. I looked around a bit more and found the source at https://www.postgresql.org/ftp/source/ -- in hindsight, very obvious.
I went through the source code from 8.4.13 up to 9.0.0 looking for "XID" in pg_controldata.c. It turns out that 9.0.0 added this field and later versions don't support migrating without it.
[pgdg90]
name=PostgreSQL 9.0 RPMs for RHEL/CentOS 6
baseurl=https://yum-archive.postgresql.org/9.0/redhat/rhel-6-x86_64
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
I added this repo to /etc/yum/repos.d and then checked the available versions with yum --showduplicates list postgresql90 -- the earliest available version was 9.0.6.
I shut down postgresql and installed 9.0.6 with yum install postgresql90-server-9.0.6-1PGDG.rhel6 postgresql90-devel-9.0.6-1PGDG.rhel6 postgresql90-libs-9.0.6-1PGDG.rhel6.
Strangely enough, pg_upgrade didn't actually exist in /usr/pgsql-9.0/bin... so a quick yum whatprovides '*pg_upgrade' | grep 9.0.6 showed me that I actually had to install postgresql90-contrib-9.0.6-1PGDG.rhel6.x86_64 as well.
I added /usr/pgsql-9.0/bin to my $PATH, ran initdb, ran pg_upgrade with --check, and all seemed well. I grabbed the settings I needed from the old postgresql.conf and pg_hba.conf and copied them to the 9.0 conf files, ran pg_upgrade, and everything went smoothly.
After checking basic operations, database integrity, and functions with the web apps we have, I went ahead and did yum update postgresql90 to update to the latest minor minor version in pgdg (9.0.23, in this case) and that went swimmingly as well.
So, if anyone else happens to find this niche problem, that's how you can try dealing with it. Best of luck!
Related
While following installation instructions (e.g., for caffe for os x), I run into the --fresh flag for homebrew. For example,
brew install --fresh -vd snappy leveldb gflags glog szip lmdb
However, I see no documentation about what --fresh does, and I don't find it in the source code for homebrew. What does this flag do? (Or what did it used to do?)
I found an old github issue describing the behavior of --fresh.
The flag was meant to ensure packages would be installed without any previously set compile-time options (like --with-python), but it was removed because it didn't do anything:
commit 64744646e9be93dd758ca5cf202c6605accf4deb
Author: Jack Nagel <jacknagel#gmail.com>
Date: Sat Jul 5 19:28:15 2014 -0500
Remove remaining references to "--fresh"
This option was removed in 8cdf4d8ebf439eb9a9ffcaa0e455ced9459e1e41
because it did not do anything.
I am currently experimenting with MongoDB replica set mechanism.
I already have a working standalone Mongo server with a main database of about 20GB of data.
I decided to convert this mongo server to a primary replica set server, then added a 2nd machine with a similar configuration (but a newer mongo version), as a secondary replica set server.
This works fine, all data is replicated to the secondary as expected.
But I would like to perform some alteration operations on the data (because somehow, my data model has changed and I need to, for example rename some properties, or convert references to a simple ObjectId, some things like that). By the same time I would like to update the first server which has an old version (2.4) to the last version available (2.6).
So I decided to follow the instructions on the MongoDB website to perform maintenance on replica set members.
shut down the secondary server. (ok)
restart server as standalone on another port (both servers usually run on 27017)
mongod --dbpath /my/database/path --port 37017
And then, the server never restarts correctly and I get this:
2014-10-03T08:20:58.716+0200 [initandlisten] opening db: myawesomedb
2014-10-03T08:20:58.735+0200 [initandlisten] myawesomedb Assertion failure _name == nsToDatabaseSubstring( ns ) src/mongo/db/catalog/database.cpp 472
2014-10-03T08:20:58.740+0200 [initandlisten] myawesomedb 0x11e6111 0x1187e49 0x116c15e 0x8c2208 0x765f0e 0x76ab3f 0x76c62f 0x76cedb 0x76d475 0x76d699 0x7fd958c3eec5 0x764329
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1187e49]
/usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x116c15e]
/usr/bin/mongod(_ZN5mongo8Database13getCollectionERKNS_10StringDataE+0x288) [0x8c2208]
/usr/bin/mongod(_ZN5mongo17checkForIdIndexesEPNS_8DatabaseE+0x19e) [0x765f0e]
/usr/bin/mongod() [0x76ab3f]
/usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x76c62f]
/usr/bin/mongod(_ZN5mongo13initAndListenEi+0x1b) [0x76cedb]
/usr/bin/mongod() [0x76d475]
/usr/bin/mongod(main+0x9) [0x76d699]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd958c3eec5]
/usr/bin/mongod() [0x764329]
2014-10-03T08:20:58.756+0200 [initandlisten] exception in initAndListen: 0 assertion src/mongo/db/catalog/database.cpp:472, terminating
2014-10-03T08:20:58.757+0200 [initandlisten] dbexit:
What am I doing wrong ?
Note that at this time, the first server is still running as primary member.
Thanks in advance!
I believe you are hitting a bug in VMWare here (can you confirm you are using VMWare VMs? confirmed) - I have seen it confirmed on Ubuntu and Fedora so far. The bug causes pieces of previous data to not be zero'ed out when creating the MongoDB namespace files (not always, but sometimes). That previous data essentially manifests as corruption in the namespace files and leads to the assertion you saw.
To work around the issue, there will be a fix released in MongoDB versions 2.4.12 and 2.6.5+ as part of SERVER-15369. The OS/Kernel level fix will eventually percolate down from the kernel bug and the Ubuntu patch, but that may take some time to actually be available as an official update (hence the need for the workaround change in MongoDB itself in the interim).
The issue will only become apparent when you upgrade to 2.6 because of additional checking added to that version that was not present in 2.4, however the corruption is still present, just not reported on version 2.4
If you still have your primary running, and it does not have the corruption, I would recommend syncing a secondary that is not on a VMWare VM and/or taking a backup of your files as soon as possible for safety - there is no automatic way to fix this corruption right now.
You can also look at using version 2.6.5 once it is released (2.6.5 rc4 is available as of writing this which includes the fix). You will still need to resync with that version off your good source to create a working secondary, but at least there will then be no corruption of the ns files.
Updates:
Version 2.6.5 which includes the fix mentioned was released on October 9th
Version 2.4.12 which includes the fix was released on October 16th
Official MongoDB Advisory: https://groups.google.com/forum/#!topic/mongodb-announce/gPjazaAePoo
I updated my DB plan on heroku quite some time ago, following this clear tutorial: https://devcenter.heroku.com/articles/upgrade-heroku-postgres-with-pgbackups
So now I have 2 DB running:
$ heroku pg:info
=== HEROKU_POSTGRESQL_NAVY_URL (DATABASE_URL)
Plan: Crane
Status: Available
Data Size: 26.1 MB
Tables: 52
PG Version: 9.2.6
Connections: 8
Fork/Follow: Available
Rollback: Unsupported
Created: 2013-11-04 09:42 UTC
Region: eu-west-1
Maintenance: not required
=== HEROKU_POSTGRESQL_ORANGE_URL
Plan: Dev
Status: available
Connections: 0
PG Version: 9.2.7
Created: 2013-08-13 20:05 UTC
Data Size: 11.8 MB
Tables: 49
Rows: 7725/10000 (In compliance, close to row limit) - refreshing
Fork/Follow: Unsupported
Rollback: Unsupported
Region: Europe
I keep receiving mails saying that I'm close to rate limit on HEROKU_POSTGRESQL_ORANGE_URL. I'd rather delete it, but I'd like to make sure I'm not going to loose any data. Heroku is not clear about it:
The original database will continue to run (and incur charges) even after the upgrade. If desired, remove it after the upgrade is successful.
But can I be 100% sure that all the data in the HEROKU_POSTGRESQL_ORANGE_URL is duplicated in the HEROKU_POSTGRESQL_NAVY_URL? Because if HEROKU_POSTGRESQL_ORANGE_URL were a follower of HEROKU_POSTGRESQL_NAVY_URL, its data should be as big as the first one.
So I just need a confirmation.
Thanks
It sounds to me like the upgrade dumped and reloaded the DB. So the new DB is a copy of the old one. If that's the case, it will contain all data from the old one at the time it was copied - but if you kept on adding new data to the old database, that data wouldn't appear in the new one.
I strongly recommend that before dropping the DB you:
Disable access to it except for pg_dump
Dump it with pg_dump (or use Heroku's tools to do that)
... and only then delete it.
That way, if you discover you've made a mistake, you have a dump to restore.
I have upgraded from 9.2 to 9.3 successfully on ubuntu. However,
/etc/init.d/postgresql start
starts both 9.2 and 9.3
Although the above command can accept that the version number and successfully starts and stops each one, is there any method I can use to make this command start 9.3 only.
The reason is that, I am not able to reboot the system now, but I am afraid when it is rebooted both servers can start.
My short term solution is to adjust the port numbers to make my application use 9.3 database. However, I would like to learn about more permanent and robust solutions.
Thanks in advance,
Steve
Ubuntu uses pg_wrapper to manage PostgreSQL installs. See the Ubuntu PostgreSQL wiki page.
You'll want to pg_dropcluster the 9.2 cluster, if you wish to actually destroy the old data. Or un-install PostgreSQL 9.2. Or modify the config file (don't remember the name right now) in /etc/postgresql/9.2/ that controls whether Pg starts or not. It's called something like start.conf or pg_ctl.conf or something.
You may also want to reverse the configured ports so your new 9.3 runs on 5432 and your not-started-by-default 9.2 tuns on 5433. That is in postgresql.conf.
Steve Harman's response worked perfectly for me, too:
Thanks for the response. In the /etc/postgresql/9.2/main/ directory, there is start.conf. If you change the single line in that file from 'auto' to 'disabled' then, /etc/init.d/postgresql start will not start 9.2. – Steve Harman Jan 1 at 16:55
On the other hand, just fyi, the command output is that both versions of the server are starting (which is not true and is coming from the service starting scripts)
user#server:/etc/postgresql/9.3/main$ sudo service postgresql start
* Starting PostgreSQL 9.1 database server
...done.
* Starting PostgreSQL 9.3 database server
...done.
I was working with postgresql, and suddenly, this stop it. I stoped service, but when i try start it, never i can do it.
service postgresql start FAIL
don't have a backup, and with pg_dump is imposible.
pg_dump -i -h localhost -p 5432 -U postgres -F c -b -v -f "/mibase.backup" mydatabase
which is the best form to do a backup?
I had a lot of trouble with PostgreSQL 9.0.2 under Windows. The service would just stop every couple of days. I never had trouble restarting it, though. Never had to restart the Windows server to restart the PostgreSQL service.
With the PostgreSQL service shut down, you can copy the database files.
If you're not on the most current release of your version, you might try installing a more recent minor version. Postgresql doesn't mind running multiple versions on the same server, although they each have to listen on a different port.
The minor version number is the third digit. Above, the major version is "9.0". If you're running 9.0.2, you want 9.0.[a number greater than 2]. Why?
"Minor releases never change the internal storage format and are always compatible with earlier and later minor releases of the same major version number, e.g., 8.4.2 is compatible with 8.4, 8.4.1 and 8.4.6." (Upgrading a PostgreSQL Cluster)
So a minor version upgrade, listening on a different port, and pointing to your old data directory might let you make a SQL dump. A SQL dump can be restored to any version.
A minor version upgrade pointing to a new data directory should be able to read files copied at the filesystem level. (Paragraph 2, way above.)