I would like to generate dump from a remote PostgreSQL database (PGAAS) with commands or Python code.
Firstly I tried locally to do the work but I have an error :
pg_dump: error: server version: 13.9; pg_dump version: 12.12 (Ubuntu 12.12-0ubuntu0.20.04.1)
I tried this code :
import subprocess
dump_file = "database_dump.sql"
with open(dump_file, "w") as f:
print(f)
subprocess.call(["pg_dump", "-Fp", "-d", "dbdev", "-U", "pgsqladmin", "-h", "hostname"-p", "32000"], stdout=f)
How can I do to have a pod (container) doing this work and where version is the same that server version, without entering pgaas password manually ?
pg_dump: error: server version: 13.9; pg_dump version: 12.12 (Ubuntu 12.12-0ubuntu0.20.04.1)
As you can see this error is caused because of a version mismatch checking the version of your PGaaS database and the database version you are using on your local machine. If your local version is lower than that of the server version you can upgrade the local version. Follow this document for upgrading your pg version.
If you want to take dumps at regular intervals in an easy way you can have a cron job scheduled on your vm for running your code. Since you want to use kubernetes, build a docker image with your code in it and create a kubernetes job and run it with kube-scheduler and you can use environment variables for encrypting your password.
Related
Trying to install cstore_fdw on a docker postgres container just to test the columnar functionality, and I'm stuck at the point of build/make the cstore repo.
The os is debian 9, postgres 11, i've installed make and a few tools I need.
The step where you build and run the make cmd, is failing bcz of postgres. Found in "building" section here: https://github.com/citusdata/cstore_fdw
Directories
cstore dir: /var/lib/postgresql/cstore_fdw/
pg config dir: /var/lib/postgresql/data/postgresql.conf
at the cstore dir I run the following and get the following error, and I don't know why.
PATH=/var/lib/postgresql/data/:$PATH make
Makefile:38: /usr/lib/postgresql/11/lib/pgxs/src/makefiles/pgxs.mk: No such file or directory
Makefile:45: *** PostgreSQL 9.3 or 9.4 or 9.5 or 9.6 or 10 or 11 is required to compile this extension. Stop.
I've also tried to use the pg bin directory in the make cmd, which is /usr/lib/postgresql/11/bin/, I think, and that gives same error.
Thanks for any help
Your docker image does not contain the PostgreSQL extension building infrastructure PGXS.
You will need to install that and the PostgreSQL header files.
I am trying to use postgres on my machine and realized that some time ago I had installed the database software after running brew install postgres and receiving Error: postgresql-9.4.5_2 already installed. From there I decided to install homebrew/services to be able to run postgres in the background and successfully launched and closed the background job with
brew services start postgresql==> Successfully startedpostgresql(label: homebrew.mxcl.postgresql)
However, when I run postgres I received the error of:
postgres does not know where to find the server configuration file.
You must specify the --config-file or -D invocation option or set the PGDATA environment variable.
When I followed an answer on SO, cat /usr/local/var/postgres/server.log I received the following error:
LOG: skipping missing configuration file "/usr/local/var/postgres/postgresql.auto.conf"
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.4.5.
Which makes me think that I had an older version of PostgreSQL installed that needs to be removed.
I then tried psql and received:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
And finally decided to init a new db with initdb /usr/local/var/postgres, but that didn't fix it, which resulted in following the original message and setting an env varible: export PGDATA=/usr/local/var/postgres. After trying postgres again, I receive
LOG: skipping missing configuration file "/usr/local/var/postgres/postgresql.auto.conf"
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.4.5.
What this tells me is that there is clearly a version compatibility issue, which needs to be resolved in order to launch my postgres server, but I'm not sure what file paths need to be cleared or the proper way to clear them. Can anyone provide some guidance?
I had the same problem and I got it to work by running:
pg_ctl -D /usr/local/var/postgres start && brew services start postgresql
(this guide was good).
I have Cloudera Manager 5.9 installed on Ubuntu 12.04 with embedded postgres database. I upgraded Ubuntu to 14.04 using do-release-upgrade. In the process, Postgres also got upgraded from 8.4 to 9.3. Now when I try to start the CM database via:
# sudo service cloudera-scm-server-db start
I get the following error in CM db.log:
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 8.4, which is not compatible with this version 9.3.15.
How do I get past this? I have looked at a lot documentation which talks about dumping the postgres database via pg_dump and restoring via psql, but I don't know how this applies in the context of cloudera manager, especially when the database is not coming up.
On Ubuntu 12.04 when everything is working, I believe the dump can be taken like this:
#pg_dump -h localhost -p 7432 -U scm > /tmp/scm_server_db_backup.$(date +%Y%m%d)
I can try to create an empty database and restore the dump to this one using psql. But how do I configure cdh to point to this database?
As you suggest, you need to find a way to "convert" 8.4 data files to 9.3 data files.
Using pg_dump will need a working PostgreSQL 8.4 instance. So, basically you need a working Postgresql 8.4 (think VM or Docker), then copy your existing 8.4 fiels to that VM/Docker, have that VM/Docker provide a plain-text dump [plain SQL, so compatible with any version), restore that plain-text dump to your 9.3 instance).
You can try :
Create a VirtualMachine or a Docker instance with Postgresql 8.4
deployed.
Locate the main data directory (usually /var/lib/postgresql/8.4/main on Ubuntu, but this might differ) of
your upgraded Cloudera machine. Backup this directory and keep in
safe.
Stop PostgreSQL on your VM/Docker if necessary. Locate the main data directory (usually /var/lib/postgresql/8.4/main on Ubuntu, but
this might differ).
Replace the previous found directory by a copy of your existing 8.4/main content (the one on your upgraded machine having now PG 9.3) to your VM/Docker.
Restart PostgreSQL 8.4 on the VM/Docker
Use pg_dumpall to create a full backup :
pg_dumpall > dump.sql
Transfer dump.sql to your Cloudera machine, and restore it. You might need to drop previous schemas/databases :
psql -f dump.sql postgres
I am able to resolve this problem using the following process:
Step 1: Take a dump of the running postgres database on Ubuntu 14.02
# sudo su
# su - postgres
# pg_dump -h localhost -p 7432 -U scm scm > scm.sql
Step 2: Upgrade Ubuntu to 16.04
# sudo do-release-upgrade
...
Step 3: Rename the old data directory
# mv /var/lib/cloudera-scm-server-db/data/ /var/lib/cloudera-scm-server-db/data9-3
Step 4: Restart cloudera-scm-server-db service. This will create an empty database which we will populate using the backup taken in step 1
# sudo service cloudera-scm-server-db restart
Step 5: Now restore the database
# sudo su
# su - postgres
# psql -h localhost -p 7432 -U scm
(password can be obtained like this: grep password /etc/cloudera-scm-server/db.properties)
scm> \i scm.sql
Step 6: Now restart cloudera-scm-server service:
# sudo service cloudera-scm-service restart
Being new to Anaconda, I am having some trouble properly setting up a conda environment. What I am interested in achieving is setting up an environment for a django application with a postgres database. The following command creates the environment:
$ conda create -n django1.7-webdev python=3.4 django=1.7 postgresql=9.1
This second command activates the environment:
$ source activate django1.7-webdev
At this point, though, when trying to run psql, I get the following error:
$ psql
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
How can I start PostgreSQL in the conda environment? The following command starts the PostgreSQL installed outside the activated conda environment, which is not what I want:
$ sudo service postgresql start
The postgresql documentation on starting servers is at https://www.postgresql.org/docs/9.1/static/server-start.html - before that, you might also need to initialize a database: https://www.postgresql.org/docs/9.1/static/creating-cluster.html
The conda package should include any binaries necessary to follow those directions. Moreover, these binaries should already be on PATH, since you are activating the environment.
In general, if you're starting a command with sudo to interact with conda, something is wrong. Unless you are trying to do some centrally-owned install that several users use, conda should never require admin rights.
I realized that PostgreSQL was already running on my laptop (Mac OS X) before I installed from the Postgres site. So when I used the installer, I got the PostgreSQL and logged in to the postgres user account that was created.
In the terminal I wrote
psql -U postgres
And provided my password. I got logged in but it said,
WARNING: psql version 9.0, server version 9.1.
Some psql features might not work.
How should I go about fixing this so that I can access the database properly without any issues?
The warning comes from psql, the PostgreSQL interactive terminal. Nothing bad will happen.
As you have two versions of PostgreSQL installed in parallel, you would need two versions of psql. Maybe you even have them on disk. But when you type the command psql, your system will default to one of those, not knowing beforehand which database server version you are going to connect to.
You can type the explicit path to the psql version you want. Find the full path of all variants with this shell command (works with Linux, not tested with Mac OS X):
which -a psql
If you did not also install the psql version 9.1 along with your PostgreSQL, you have to install it first, of course.
If you are not going to use PostgreSQL 9.0 any more, you can uninstall it to remove ambiguities.
In Debian you can also set the default of multiple alternatives with:
update-alternatives
But in Debian you also have a wrapper that calls the matching psql dynamically if you specify the database cluster like this:
psql --cluster 9.1/main
Not sure about Mac OS X.
You have installed postgresql-server 9.1 (server side) and postgres (client side 9.0). Maybe you have installed client 9.1 too, but it is not on the path, so you have to find it or if you have not it, then install it.