I want to use pg_dump to create a custom format dump (--format=custom) from Google Cloud SQL, then import that into my local database for development. My attempts to import the dump result in the error:
pg_restore: [custom archiver] unrecognized data block type (151) while searching archive
My DuckDuckGo-fu yields no relevant results. There is a similar post for exporting in plain format, but plain format is about 10 times the size of a custom archive format. When I import a plain SQL file, it is successful.
There is documentation for doing the reverse, Exporting and importing using pg_dump and pg_restore, but this is not what I want to do.
Finally, I am able to import custom archives in other projects I work on, including one on AWS and another on Digital Ocean, so I wonder what I'm doing wrong on Google Cloud SQL.
Edit
Here's the pg_dump command.
pg_dump --format=c --no-owner --no-acl --file="~/projects/project/backups/project-custom.sqlc" --dbname=project --schema='public'
I have no clue how to do a pg_restore of this.
Related
Can someone help me, please? I need to perform pg_dump from DB and that must be read by Athena
pg_dump --> s3 < ---- aws athena
How can I do this?
The PostgreSQL pg_dump format is pure SQL that can be run on a PostgreSQL database to create tables and load data. Open it in a text editor and take a look -- you'll see what I mean.
As a result, pg_dump files are not in a format that can be used with Amazon Athena.
I downloaded the postgresql .dmp file from the chembl database.
I want to import this into gcp cloudsql.
When I run it with the console and gcloud command, I get the following error:
Importing data into Cloud SQL instance...failed.
ERROR: (gcloud.sql.import.sql) [ERROR_RDBMS] exit status 1
The input is a PostgreSQL custom-format dump.
Use the pg_restore command-line client to restore this dump to a database.
Can I import custom-format dmp files without using the pg_restore command?
https://cloud.google.com/sql/docs/postgres/import-export/importing
There is a description of pg_restore in the document on that site, but I didn't get it right.
In the case of custom-format files, is it necessary to pg_restore after uploading them to the cloud shell?
According to the CloudSQL docs:
Only plain SQL format is supported by the Cloud SQL Admin API.
The custom format is allowed if the dump file is intended for use with pg_restore.
If you cannot use pg_restore for some reason, I would spin up a local Postgres instance (i.e., on your laptop) and use pg_restore to restore the database.
After loading into your local database, you can use pg_dump to dump to file in plaintext format, then load into CloudSQL with the console or gcloud command.
I need to understand the relations between tables in a PostgreSQL database. I will not have the ability to download pgAdmin4 like I am used to working in. So after looking around I found pg_dump.exe built into PostgreSQL. I thought I could just do something like this in the SQL Shell (psql) to get a dump of the database and then have the ability to upload it into another system:
database=# pg_dump database > path/to/save/file.sql;
Based on the docs >> https://www.postgresql.org/docs/9.3/app-pgdump.html
But when I run that I get an error:
ERROR: syntax error at or near "pg_dump"
LINE 1: pg_dump database > path/to/save/file.sql;
^
I saw on this stack overflow question was similar to my issue with the pg_dump error. In the solutions, Adrian says that pg_dump does not work within psql. When I run the code that Adrian suggested, I also get an error.
Any thoughts on how I can use psql to get the information that I need of this database?
Note: I am accessing a Linux VM on a Windows machine.
I'm trying to export my database created in Google Cloud Sql and import it into a new external server.
I tried to create a sql backup through the google console, downloaded it and copied it to the new server via filezilla and then launched the following command:
psql -U postgres -d ciclods-db -1 -f Backup-db_Cloud_SQL_Export_2019-03-23\ \(17_01_19\)
but i get this output:
ERROR: role "cloudsqladmin" does not exist
REVOKE
ERROR: role
"cloudsqlsuperuser" does not exist GRANT
what is the right procedure to follow in these cases?
I have resolved the same problem by locating and deleting the two lines from the exported sql file with "cloudsqladmin". My app does not use it anyway.
to do this task you can follow the official GCP guide about How to export data from Cloud SQL[1] in that document they give you the option to export the data into a dump file or csv files which can be used for other tools.
https://cloud.google.com/sql/docs/mysql/import-export/exporting
In order to create the export file, you have to do it from a command line and use additional flags. As per documentation‘s “Exporting data to a SQL dump file”, there is a section on Exporting data from an externally-managed database server.
As well you can find there the option to export the data into a CSV file.
I made a backup of a database of postgres, it is not the first time I do it, I used this command:
pg_dump db -f /backup/agosto_31.sql
And I do the restore with this:
psql -d August_31 -f August_31.sql
But this time I did not import any trigger, and there are many. I checked in the file August_31.sql and they are. How could I import them again? Only the triggers.
Thanks everyone, greetings!
There are not any possibility to import only triggers. But, because your dump (backup file) is in SQL format (plain text), you can cut trigger definition manually in any editor. For this case is practical dump data and schema to separate files.
Is possible so import (restore from backup) fails, probably due broken dependencies. Check psql output if there are some errors. psql has nice possibility to stop on first error:
psql -v ON_ERROR_STOP=1
Use this option. Without error specification is not possible to help more.