I had a backup script for a small server running Postgresql in a Docker Container.
The backup line was the following:
docker exec -t crm-database-1 pg_dump --dbname=postgresql://postgres:**************#localhost:5432/********** -F c -b -Z 9 -E UTF8 -v > $path/db.bak
Today the unthinkable happened and the Postgres data directory got corrupted. No chance of recovering.
I think I screwed myself because of the -v flag.
And of course I never tried to recover my backups before.
Now that im trying to restore one of the 20 Dumps I have but I only get errors like "unexpected EOF" or "file is not an archive"
Is there any chance on recovering or did I screw up and the data is gone?
I tried every possible combination of piping the backup to pg_restore or pg_ctl.
Also tried to edit the backup file by removing all "pg_dump: reading table..." with regex.
None did work.
EDIT:
After a long search i found out after "cat-ing" out something from docker to host, the file that gets created on the host, has a few bytes more.
With HxD I found, there are extra 0D (LineBreak) characters.
After removing them I was able to restore the schema but when trying to import data, I get the error message "out of memory"
Im going to dig a little bit more into that.
Related
Using pg_dump and pg_restore on postgres to move a dbs from my local Windows machine to a Linux server seemed so simple:
pg_dump --format=c -U user localdbs > file.pg.dump
Spits out a file. Then upload:
pg_restore -c -d serverdbs -v file.pg.dump
pg_restore: error: input file does not appear to be a valid archive
This works perfectly on Linux to Linux. Changing the filetype makes no difference. Changing the encoding at either end makes no difference. Pulling your hair out makes no difference.
It's the > operator in the pg_dump command. It looks like it works on Windows, it spits out a file but that file is not properly encoded. On Linux the same command works flawlessly.
You have to use -f file.pg.dump instead when using Windows and then everything works.
Hope this saves someone the nightmare I had figuring this out.
My first post here may go a bit against the grain - if it does violate guidelines feel free to delete. The problem itself has been solved, but since it was quite frustrating and the solution is documented nowhere, however simple it is, I thought I post here for fellow devs encountering the same. Beyond that, if someone can shed some light as to why this happened in the first place, Answers are more than welcome.
The Issue
Downloading data from local PostgreSQL instance using pg_dump from a DB1
Full command: pg_dump -U {user} -d {sourceDB} -t {sourceTable} > {dump}.sql
Uploading the data on the same server in a different DB2 using psql
Full command: psql -U {user} -d {targetDB} -f {dump}.sql
On Import, the following error is thrown
psql:tbl_strainsLog.sql:1: ERROR: syntax error at or near "ÿ_" LINE 1: ÿ_-
^
While the error may seem intuitive, there is no such character anywhere in the file.
I had tried multiple different options for export and the import, all throw the same issue.
Encoding of source & target DB were set to UTF8, collation was en_US.utf8 on both as well.
I checked the initially loaded data, encoding was UTF8 there as well.
SETUP
Windows 10
Powershell
Postgres 13
Thanks, and I a m very curious if anyone has deeper insights as to what causes this issue in the first place.
So the culprit for the issue seems Powershell itself. While I have no exact knowledge of why this would happen, bypassing Powershell did the trick.
If I either run the above commands from cmd or I "pipe" thepg_dump command through cmd.exe, the error disappears and it works flawlessly.
So either...
A) in cmd run pg_dump -U {user} -d {sourceDB} -t {sourceTable} > {dump}.sql
or
B) in Powershell run & cmd.exe /c "pg_dump -U {user} -d {sourceDB} -t {sourceTable} > {dump}.sql"
This resolves the encoding issue created by pure Powershell during the export/dump.
The import works fine in Powershell, so the encoding issue only applies during export.
Hope this helps people struggeling with the same.
I have a scenario where a certain data set comes from a CSV and I need to allow a non-dev to hit PG Admin and update this data set. I want to be able to put this CSV in a mapped folder from the host system and then use the PG Admin GUI to run a COPY command. So far PG Admin is telling me:
ERROR: could not open file "/var/lib/pgadmin/data-files/some_data.csv" for reading: No such file or directory
Here are my steps so far, along with a sanity check inspect:
docker volume create --name=data-files
docker run -e PGADMIN_DEFAULT_EMAIL="pgadmin#example.com" -e PGADMIN_DEFAULT_PASSWORD=some_pass -v data-files:/var/lib/pgadmin/data-files -d -p 5050:80 --name pgadmin dpage/pgadmin4
docker volume inspect data-files --format '{{.Mountpoint}}'
/app/docker/volumes/data-files/_data
docker cp ./updated-data.csv pgadmin:/var/lib/pgadmin/data-files
And, now I think that PG Admin could see the updated-data.csv, so I try COPY, which I know works locally on my dev system where PG Admin is on baremetal:
COPY foo.bar(
...
)
FROM '/var/lib/pgadmin/data-files/updated-data.csv'
DELIMITER ','
CSV HEADER
ENCODING 'windows-1252';
Is there any glaring mistake here? When I do docker cp there's no feedback to stdout. No error, no mention of success or a hash or anything.
It looks like you thought the file should be inside the pgadmin container however the file you are going to copy must be inside the postgres container so the query you run will find the file. I suggest you copy the file to postgres container :
docker cp <path_from_your_local>/file.csv <postgres_container_name>:/file.csv
Then in the query tool from your pgadmin you can copy without problems !
I hope this help to others came here...
I have taken a database backup from the server using:
pg_dump -U postgres -d machine -s --disable-triggers >aftrn.sql
When I am trying to restore that data into my local using:
psql -U postgres -d usernet < aftrn.sql
I am getting the following error of:
invalid command \N
invalid command \N
invalid command \N
out of memory
root#lap-DB3:/home/akhiles#
Does anyone know what's the reason or error related to?
When I've had this issue, it is because it's been loading into the wrong database.
Pipe the output to a file and read the first few lines of the output.
I'd like to copy the content of my local machine to my remote one (inside a docker).
For some reason, it is more complicated that I was expected:
When I try to copy the data to the remote one, I get this "ERROR: CREATE DATABASE cannot run inside a transaction block".
Ok... So I get into my docker container, added the rule \set AUTOCOMMIT inside. But I still get this error.
This is the command I did:
// backup
pg_dump -C -h localhost -U postgres woof | xz >backup.xz
and then in my remote computer:
xz -dc backup.xz | docker exec -i -u postgres waf-postgres psql --set ON_ERROR_STOP=on --single-transaction
But each time I get this "CREATE DATABASE cannot run inside a transaction block" no matter what I try. Even if I put the autocommit to "on".
Here my problem: I don't know what a transaction block is. And I don't understand why copying one db to another need to be so hard pain: My remote db is empty. So why there is so much fuss and why psql just can't force what I want?
My aim is just to copy my local db to the remote one.
what happens here is: you add CREATE DATABASE statement with -C key and then try to run psql with --single-transaction, so the content of script are wrapped to BEGIN;...END;, where you can't use CREATE DATABASE
So iether remove -C and run psql against existing database, or remove --single-transaction for psql. Make decision based on what you really need...
from man pg_dump:
-C
--create
Begin the output with a command to create the database itself and reconnect to the created database. (With a script of this
form, it doesn't matter which database in the destination installation
you connect to before
running the script.) If --clean is also specified, the script drops and recreates the target database before reconnecting to
it.
from man psql:
--single-transaction
This option can only be used in combination with one or more -c and/or -f options. It causes psql to issue a BEGIN command
before the first such option and a COMMIT command after the last one, thereby wrapping all the commands into a single
transaction. This ensures that either all the commands complete successfully, or no changes are applied.