In order to track changes for a database I thought that the best solution is to generate a snapshot of the database, make the changes and the get a changelog comparing the snapshot with the db.
For the snapshot I use the following command:
$ liquibase --url=jdbc:postgresql://localhost:5432/test
outputFile=sdk/workspace//output.json --snapShotFormat=json
Now that I have the snapshot, I made some changes in the database (adding 2 columns to 2 different tables). I try to run to following command to compare the snapshot with the db:
$ liquibase --url=jdbc:postgresql://localhost:5432/test -
username=postgres --password=new_password --
referenceUrl=offline:postgresql?snapshot=sdk/testsnapshot.json
diffchangelog
but I get the following error:
$ Unexpected error running Liquibase: Cannot parse snapshot
offline:postgresql?snapshot=sdk/testsnapshot.json
Any idea on how to fix this?
I manage to solve the problem and in case that someone will have the same issue in the future, this is the correct way to do it.
Generate the snapshot:
liquibase
--driver=org.postgresql.Driver \
--classpath=lib/postgresql42.2.5.jre6.jar \
--url=jdbc:postgresql://localhost:5432/test \
--outputFile=sdk/snaptest.json \
--username=postgres \
--password=new_password \
snapshot
Generate the diffChangeLog between the snapshot and the database
liquibase
--driver=org.postgresql.Driver \
--changeLogFile=sdk/workspace/difference_log.xml \
--url=jdbc:postgresql://localhost:5432/test \
--username=postgres \
--password=new_password \
--referenceUrl=offline:postgresql=sdk/snaptest.json \
diffchangelog
Cheers
Related
As soon as my EMR-Cluster was ready to be run.
I started facing some issues when listing databases and importing sqoop
Apparently, sqoop has been installed normally and it is working normally when I type "sqoop help" in Linux terminal.
using sqoop help
as you can see, the command could be recognized normally.
However, if I try out the sqoop import command, this one cannot be and it faces an error:
sqoop import \
--connect jdbc:postgres://sportsdb.cxri########.us-east-2.rds.amazonaws.com/SportsDB \
--username postgres \
--password mypassword \
--table addresses --target-dir s3://sqoop-table-from-rds-to-s3/sqoop-table/ -m 1 --fields-terminated-by '\t' --lines-terminated-by ','
sqoop import
The same goes to the second one, which is "sqoop list-databases" as shown below:
sqoop list-databases \
--connect jdbc:postgres://sportsdb.cxri########.us-east-2.rds.amazonaws.com \
--username postgres \
--password mypassword
sqoop list-databases
they don't really works and anything happens ;/
I also downloaded jar and put into /usr/lib/sqoop/lib/ where is the jar files on sqoop
To do it I run these two follow commands:
1) wget -O postgresql-jdbc.jar https://jdbc.postgresql.org/download/postgresql-42.3.1.jar
2) sudo mv postgresql-jdbc.jar /usr/lib/sqoop/lib/
Jar file added to sqoop/lib
Someone else has a suggestion about what can be done in order to fix this issue?
The issue could be solved after following a tip received about a typo.
Then, I just changed the word postgres for postgresql as follows:
sqoop list-databases \
--connect jdbc:postgresql://sportsdb.cxri########.us-east-2.rds.amazonaws.com \
--username postgres \
--password mypassword
That is it. The issue was fixed just adjusting something pretty simple
I have problems fetching MariaDB. Because I don't need this package I'm trying to remove it. First, I tried to understand what include it:
$ grep -nrw ../layers/ -e mariadb
Binary file ../layers/meta-openembedded/.git/index matches
../layers/meta-openembedded/meta-oe/recipes-core/packagegroups/packagegroup-meta-oe.bb:99: leveldb libdbi mariadb mariadb-native \
Looking into packagegroup-meta-oe.bb I found:
RDEPENDS_packagegroup-meta-oe-dbs ="\
leveldb libdbi mariadb mariadb-native \
mysql-python postgresql psqlodbc rocksdb soci \
sqlite \
${#bb.utils.contains("DISTRO_FEATURES", "bluez4", "mongodb", "", d)} \
"
hence I tried to remove packagegroup-meta-oe-dbs in my <image>.bb:
IMAGE_INSTALL_remove = "packagegroup-meta-oe-dbs"
But it still insists to build it.
Where is my fault?
Since packagegroup-meta-oe-dbs is a runtime dependency of packagegroup-meta-oe-dbs, you cannot remove it without removing packagegroup-meta-oe-dbs.
What you need to do is create bbappend for packagegroup-meta-oe-dbs, and add the following line to it:
RDEPENDS_packagegroup-meta-oe-dbs_remove = "mariadb"
I've generated a PostgreSQL script that I want to use to restore a database. When I go to my backup server to try to restore, I get the error: syntax error at or near "\".
It's getting stuck on the following characters \.
These appear like this:
COPY admin.roles (role_id, role_name, is_role_auto) from stdin;
\.
What's wrong with this statement? Is there config I missed? I'm on PostgreSQL 11.4 on Windows, the backup was taken with pg_dump, and I restore it using pgAdmin.
You cannot use pgAdmin to restore a "plain format" dump taken with pg_dump. It doesn't understand the psql syntax where COPY and its data are interleaved.
You will have to use psql to restore the dump:
psql -h server -p 5432 -U user -d database -f dumpfile.sql
It's really hard to know the specific error without seeing your backup and restore commands in their entirety, but if it helps, here is the boilerplate I use when I want to copy a table from production to a backup server:
$BIN/pg_dump -h production_server -p 5432 \
--dbname=postgres \
--superuser=postgres \
--verbose \
--format=c \
--compress=9 \
--table=admin.roles > backup.sql
$BIN/pg_restore \
--host=backup_server \
--port=5432 \
--username=postgres \
--dbname=postgres \
--clean \
--format=custom \
backup.sql
The format=c (or --format=custom) makes the content completely unreadable, but on a plus side it also avoids any weird errors with delimiters and the like, and it also perfectly copies complex data structures like arrays and BLOBs.
I have take backup by pg_dumpall > test.out
and test.out successfully generated, hence backup completed.
I have used command psql -f test.out postgres for restore
But got following errors with restoring backup:
databases already exists
relation "products" already exists
duplicate key value violates unique constraint "products_pkey"
I actually want to replace the data in the existing db with backup. How to do that?
The problem is that the database you're trying to restore already exists.
You can run a DROP DATABASE database_name command that will delete your existing database and then you can run your test.out file.
Or you can run pgdumpall --clean > test.out and then run the resulting file. The clean flag will make the resulting files have the DROP DATABASE command in them.
Do you use the bellow command ?
psql -h localhost -U [login role] database_name -f /home/database.backup
I think a flow like this might help, because we don't want drop the database each time we call the backup file.
First, we need to create a backup file using the --format=custom [-Fc] to restore it using pg_restore. We can use a connection string postgresql://<user>:<pass>#localhost:5432/<dbname> and replace <user>, <pass>, and <dbname> with your information.
pg_dump -v -Fc \
postgresql://<user>:<pass>#localhost:5432/<dbname> \
> db-20211122-163508.sql
To restore we will call it using --clean [-c] and --create [-C] to drop the database before restoring. Replace <user>, <host>, <port>, and <dbname> with your information.
pg_restore -vcC \
-U <user> \
-h <host> \
-p <port> \
-d <dbname> \
< db-20211122-163508.sql
This way you don't need to use clean when you create the backup file.
I'm trying to port a database from MySQL to PostgreSQL. I've rebuilt the schema in Postgres, so all I need to do is get the data across, without recreating the tables.
I could do this with code that iterates all the records and inserts them one at a time, but I tried that and it's waaayyyy to slow for our database size, so I'm trying to use mysqldump and a pipe into psql instead (once per table, which I may parallelize once I get it working).
I've had to jump through various hoops to get this far, turning on and off various flags to get a dump that is vaguely sane. Again, this only dumps the INSERT INTO, since I've already prepared the empty schema to get the data into:
/usr/bin/env \
PGPASSWORD=mypassword \
mysqldump \
-h mysql-server \
-u mysql-username \
--password=mysql-password \
mysql-database-name \
table-name \
--compatible=postgresql \
--compact \
-e -c -t \
--default-character-set=utf8 \
| sed "s/\\\\\\'/\\'\\'/g" \
| psql \
-h postgresql-server \
--username=postgresql-username \
postgresql-database-name
Everything except that ugly sed command is manageable. I'm doing that sed to try and convert MySQL's approach to quoting single-quotes inside of strings ('O\'Connor') o PostgreSQL's quoting requirements ('O''Connor'). It works, until there are strings like this in the dump: 'String ending with a backslash \\'... and yes, it seems there is some user input in our database that has this format, which is perfectly valid, but doesn't pass my sed command. I could add a lookbehind to the sed command, but I feel like I'm crawling into a rabbit hole. Is there a way to either:
a) Tell mysqldump to quote single quotes by doubling them up
b) Tell psql to expect backslashes to be interpreted as quoting escapes?
I have another issue with BINARY and bytea differences, but I've worked around that with a base64 encoding/decoding phase.
EDIT | Looks like I can do (b) with set backslash_quote = on; set standard_conforming_strings = off;, though I'm not sure how to inject that into the start of the piped output.
Dump the tables to TSV using mysqldump's --tab option and then import using psql's COPY method.
The file psqlrc and ~/.psqlrc may contain SQL commands to be executed when the client starts. You can put these three lines, or any other settings you would like in that file.
SET standard_conforming_strings = 'off';
SET backslash_quote = 'on';
SET escape_string_warning = 'off';
These settings for psql combined with the following mysqldump command will successfully migrate data only from mysql 5.1 to postgresql 9.1 with UTF-8 text (Chinese in my case). This method may be the only reasonable way to migrate a large database if creating an intermediate file would be too large or too time consuming. This requires you manually migrate the schema, since the two database's data types are vastly different. Plan on typing out some DDL to get it right.
mysqldump \
--host=<hostname> \
--user=<username> \
--password=<password> \
--default-character-set=utf8 \
--compatible=postgresql \
--complete-insert \
--extended-insert \
--no-create-info \
--skip-quote-names \
--skip-comments \
--skip-lock-tables \
--skip-add-locks \
--verbose \
<database> <table> | psql -n -d <database>
Try this:
sed -e "s/\\\\'/\\\\\\'/g" -e "s/\([^\\]\)\\\\'/\1\\'\\'/g"
Yeah, "Leaning Toothpick Syndrome", I know.