Been trying to pull my remote db into local env with heroku pg:pull, prior without problems, but at this point the command is returning this:
/bin/sh: pg_dump: command not found
▸ pg_dump errored with 127
pg_restore: [custom archiver] could not read from input file: end of file
looked for an answer here and updated the bash profile as some topics mention:
export PATH=/Applications/Postgres.app/Contents/Versions/10/bin:$PATH
rebooted and restarted terminal, but with no avail. anyone knows what's going on?
Ypsql tries to interpret SQL queries while you are adding it a compressed dump (heroku use this).
When You can not read the dump, pg_restore -O latest.dump gives you valid SQL you may pipe to psql
the easiest here :
pg_restore -O -d app_development latest.dump
Imporatant:
Try Use -O you may not use the random username of your remote heroku postgres db.
Check this link for more help here stackoverflow.com/a/11391586/6392696
Update your Heroku CLI; the issue where Postgres.app paths weren't being handled correctly was just fixed. https://github.com/heroku/cli/pull/1074
Related
command I am trying to execute:
pg_ctl init -D <path-to-db>
The error its giving me:
The files belonging to this database system will be owned by user "aashishgangwani".
This user must also own the server process.
initdb: error: file "/opt/homebrew/Cellar/libpq/13.3/share/postgresql/postgres.bki" does not exist
This might mean you have a corrupted installation or identified
the wrong directory with the invocation option -L.
pg_ctl: database system initialization failed
When trying "which postgresql" and "where postgresql" it gives "not found" over here:
(base) aashishgangwani#Aashishs-MacBook-Pro ~ % which postgresql
postgresql not found
(base) aashishgangwani#Aashishs-MacBook-Pro ~ % where postgresql
postgresql not found
Update: I have tried doing brew reinstall postgres it doesn't work
I did some debugging and found out that I may need several files of postgres into libq share folder(/opt/homebrew/Cellar/libpq/13.3/share/) so I copy pasted the whole postgres directory(/opt/homebrew/Cellar/postgresql/13.3/share/postgresql) into the libq share folder(/opt/homebrew/Cellar/libpq/13.3/share/) and pg_ctl init -D <path-to-db>
is working fine for me now.
I have installed Postgres 12 on Ubuntu by building it from source and I am facing two issues:
Although I followed the installation manual from Postgrez, every time I restart my computer, my Postgres server stopz and is no longer seen as a running process.
To start it the first time after install, I do this from the terminal:
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start
After a restart, to start DB again when I run: /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data, it throws this error:
initdb: error: directory "/usr/local/pgsql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/usr/local/pgsql/data" or run initdb
with an argument other than "/usr/local/pgsql/data".
Does that mean that every time I start Postgres after a restart, I have to create a new /data directory?
Upon installing Postgres sing pip or pip3, one can just switch user to postgres and run psql to enter postgres, however now I have to run "/usr/local/bin/psql". Please note I have exported all the paths per https://www.postgresql.org/docs/12/installation.html. How can I fix this? Can an alias be set for this?
After a restart, to start DB again when I run:
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data, it throws this
error:
Does that mean that every
time I start Postgres after a restart, I have to create a new /data
directory?
No, quite the opposite. You don't need to initdb after the first time, you just need to start. It is your attempt to initdb when you don't need to which is causing the error message. Note that attempting to initdb isn't doing any harm, because it refused to run. It just generates log/console noise.
Upon installing Postgres sing pip or pip3, one can just switch user to
postgres and run psql to enter postgres, however now I have to run
"/usr/local/bin/psql". Please note I have exported all the paths per
https://www.postgresql.org/docs/12/installation.html. How can I fix
this?
I don't know what your first sentence means, as you don't use pip or pip3 to install PostgreSQL (or at least, the docs don't describe doing so) although you might use them to install psycopg2 to enable python to talk to PostgreSQL.
You could use an alias, but it would probably make more sense to edit ~/.bash_profile to set the PATH, as described from the page you linked to under Environment Variables.
You have to register postgreSQL as a service.
run this:
pg_ctl register [-N servicename] [-U username] [-P password] [-D datadir] [-S a[uto] | d[emand] ] [-w] [-t seconds] [-s] [-o options]
Example:
pg_ctl register -N postgresql -U OS_username -P OS_password -D '/etc/postgresql/12/data' -w
More info in the manual: pg_ctl
Notes:
Username and Password is related to the OS, not postgresql
If you have doubts read the manual.
/usr/local/pgsql/bin/pg_ctl start -D '/usr/local/pgsql/data'
Export following in postgres user account's ~/.bashrc:
LD_LIBRARY_PATH=/usr/local/pgsql/lib
export LD_LIBRARY_PATH
PATH=/usr/local/pgsql/bin:$PATH
export PATH
I installed Homebrew and used it to install GDAL from within Ubuntu. After the installation finished, I'm still running into the following error: "'ogr2ogr/ogrinfo/gdalinfo' is not recognized as an internal or external command, operable program, or batch file" anytime I try using ogr2ogr, gdalinfo --version, or ogrinfo --help in command prompt. If I run those in Ubuntu, the error message reads "command not found."
Does this mean the installation did not work correctly? Would you advise to try reinstalling GDAL?
(The end goal is to use ogr2ogr to import a shapefile into postgres, eventually running
ogr2ogr -f "PostgreSQL" PG:"host=myhost user=myloginname dbname=mydbname password=mypassword" mytabfile.tab as advised by the documentation.)
If you're simply trying to import shapefiles into PostgreSQL, you might wanna take a look at shp2pgsql. It's quite simple - if you have the luxury of choosing the import tool :)
Data sample: TM_WORLD_BORDERS_SIMPL-0.3.zip
After unpacking your zip file just execute the following line in your console:
$ shp2pgsql -I -s 4326 TM_WORLD_BORDERS_SIMPL-0.3.shp table_world | psql -d mydb
Things to take into account:
table_world is the name of the target table
psql -d mydb takes into account that your current operating system user has an account in the database, that no password is required, that the database is installed at localhost and that it listens at the the standard port 5432. Check the psql documentation to build your own connection command, e.g. psql -U myuser -h 192.168.1.42 -p 5434 -d mydb to login with the user myuser in the database mydb in the remote PostgreSQL at 192.168.1.42 that listens at the port 5434. In case your PostgreSQL isn't configured to accept connections, check this answer.
4326 is the identifier for WGS84, which is the spatial reference system of this shapefile - and the most often used worldwide.
.. and your data is ready to be played with, e.g. import into QGIS:
Further reading: psql, shp2pgsql tutorial
I'm using heroku and I want to download the database from my app(heroku) so I can make some changes in it, I've installed pgbackups, but using heroku pgbackups:url downloads a .dump file
How can I download a postgresql file or translate that .dump into a postgresql file?
If you're using Heroku's pgbackups (which you probably should be using):
$ heroku pg:backups capture
$ curl -o latest.dump `heroku pg:backups public-url`
"Translate" it into a postgres db with
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
See https://devcenter.heroku.com/articles/heroku-postgres-import-export
There's a command for this in the CLI - heroku db:pull which will do this for you. db:pull can be a bit slow mind you so you may be better to use the next option.
If you are using complex postgress data types (hstore, arrays etc) then you need to use the pgtransfer plugin https://github.com/ddollar/heroku-pg-transfer which will basically does a backup on Heroku and a restores it locally.
UPDATE: db:pull and db:push have been deprecated and should be replaced with pg:pull and pg:push - read more at https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
I found the first method suggested in the documentation pull/push even easier. No password or username needed.
pg:pull
pg:pull can be used to pull remote data from a Heroku Postgres
database to a database on your local machine. The command looks like
this:
$ heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi
This command will create a new local database named “mylocaldb” and
then pull data from database at DATABASE_URL from the app “sushi”. In
order to prevent accidental data overwrites and loss, the local
database must not exist. You will be prompted to drop an already
existing local database before proceeding.
At first I had an error: /bin/sh: createdb: command not found; which I solved following this SO post.
An alternative described also in the documentation (I did not try it yet) is:
To export the data from your Heroku Postgres database, create a new
backup and download it.
$ heroku pg:backups:capture
$ heroku pg:backups:download
Source: Importing and Exporting Heroku Postgres Databases with PG Backups
To export the data from Heroku Postgres database, just follow below steps
Login to heroku
Go to APP->settings->reveal config variable
Copy DATABASE_URL
run pg_dump --DATABASE_URL_COPIED_IN_STEP_3 > database_dump_file
Note this will provide postgresql file or for dump file you can download directly from postgres addon interface.
I think the easiest way to download and replicate the database on local server:
**PGUSER**=LOCAL_USER_NAME PGPASSWORD=LOCAL_PASSWORD heroku pg:pull --app APP_NAME HEROKU_POSTGRESQL_DB_NAME LOCAL_DB_NAME
Go through this document for more info:
https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull
This is the script that I like to use.
namespace :heroku do
desc "Import most recent database dump"
task :import_from_prod => :environment do
puts 'heroku run pg:backups capture --app APPNAME'
restore_backup 'APPNAME'
end
def path_to_heroku
['/usr/local/heroku/bin/heroku', '/usr/local/bin/heroku'].detect {|path| File.exists?(path)}
end
def heroku(command, site)
`GEM_HOME='' BUNDLE_GEMFILE='' GEM_PATH='' RUBYOPT='' #{path_to_heroku} #{command} -a #{site}`
end
def restore_backup(site = 'APPNAME')
dump_file = "#{Rails.root}/tmp/postgres.dump"
unless File.exists?(dump_file)
pgbackups_url = heroku('pg:backups public-url -q', site).chomp
puts "curl -o #{dump_file} #{pgbackups_url}"
system "curl -o #{dump_file} '#{pgbackups_url}'"
end
database_config = YAML.load(File.open("#{Rails.root}/config/database.yml")).with_indifferent_access
dev_db = database_config[Rails.env]
system "pg_restore -d #{dev_db[:database]} -c #{dump_file}".gsub(/\s+/,' ')
puts
puts "'rm #{dump_file}' to redownload postgres dump."
puts "Done!"
end
end
I have a local psql database dump that needs to be uploaded to heroku. I followed the steps in the Update section from this link. Everything worked fine until the last part, the actual uploading step:
heroku pgbackups:restore --app myAppName DATABASE
'https://www.dropbox.com/myAppPSQLDumpLink/myAppName_local.dump' --confirm myAppName
This was what was showing up in the console:
HEROKU_POSTGRESQL_SILVER_URL (DATABASE_URL) <---restore--- myAppName_local.dump
Retrieving... done
! An error occurred and your restore did not finish.
And this was the error from the logs (courtesy of Toby Hede's question):
2013-01-09T15:39:09+00:00 app[pgbackups]: Invalid dump format: /tmp/GgUz5yU4bF/project_mgr_development_local.dump: HTML document text
I tried searching for this error, but could not find an answer. Does anyone know what needs to be done to solve the problem? The actual dump for my local psql database was performed this way:
pg_dump -Fc --no-acl --no-owner -U myUserName > myAppName_local.dump
Thank you!
Looks like the link to the dump on Dropbox is redirecting or pointing to an HTML page (HTML document text in the error). Visit the link and make sure you are directly getting the dump. Or download the dump in your browser, right-click on it and Copy Download Link. That link should work with pgbackups:restore.
Dropbox provides an explanation for directly downloading files (https://www.dropbox.com/en/help/201)
This can be helpful for using the dropbox links in pg_backups.
In short it says to have the download link with option "dl=1" rather than "dl=0". But this didn't work for me. Even copying the the address of downloaded file didn't work for me.
If you face the above problems, try moving the file into the public folder and copy the link from there. This worked for me.
According to Importing and Exporting Heroku Postgres Databases with PG Backups, you can restore dump at terminal with:
$ curl -o latest.dump `heroku pgbackups:url --app heroku_appname`
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
Got the same error, but different reason, so different solution. Maybe it helps someone.
If you've stored the dump file in a server with HTTPS, and you mistakenly used HTTP for the database URL, the forwarding will be interpreted as an HTML document.
So change
heroku pgbackups:restore --app myAppName DATABASE 'http://www.example.com/my.dump' --confirm myAppName
to
heroku pgbackups:restore --app myAppName DATABASE 'https://www.example.com/my.dump' --confirm myAppName