A TYPO3 installation has 57 tables in it's database named typo3.
Creating a dump using the mysqldump program by the command
mysqldump --host=127.0.0.1 --password=<PASSWORD> --protocol=tcp --port=3306 --user=<ROOT-USER> --lock-all-tables --databases typo3 > dump.sql
contains only 47 tables.
The same result occurs if the database connection is done via socket and also if the "--lock-all-tables" option is left out.
How to make a complete dump containing all the tables?
The missing tables are
index_config
index_debug
index_fulltext
index_grlist
index_phash
index_rel
index_section
index_stat_search
index_stat_word
index_words
I'm not sure, why all the index_* tables are not dumped with mysqldump, but I can suggest you to use SypexDumper. In most of the cases, when mysqldump or phpMyAdmin dumps were not possible, it saved my day.
I use this for dumping: mysqldump -u server_dbuser -p -h localhost server_db > dbdump.sql
Have you tried from phpmyadmin?
Are the index_ tables needed, or can the be rebuilt anytime anyway?
Thanks to Urs and Viktor Livakivskyi for their intention to help! The problem was caused by an individual script triggered by cron.
Related
I am using an official postgres v12 docker image that I want to initialise with two SQL dump files that are gathered from a remote legacy v9.2 postgres server during the docker build phase:
RUN ssh $REMOTE_USER#$REMOTE_HOST "pg_dumpall -w -U $REMOTE_DB_USER -h localhost -p $REMOTE_DB_PORT --clean --globals-only -l $REMOTE_DB_NAME" >> dump/a_globals.sql
RUN ssh $REMOTE_USER#$REMOTE_HOST "pg_dump -w -U $REMOTE_DB_USER -h localhost -p $REMOTE_DB_PORT --clean --create $REMOTE_DB_NAME" >> dump/b_db.sql
By placing both a_globals.sql and b_db.sql files into the docker image folder docker-entrypoint-initdb.d, then the database is initialised with the legacy SQL files when the v12 container starts (as described here). Docker is working correctly, the dump files are retrieved successfully. However I am running into problems initialising the container's database and require guidance:
When the container starts to initialise its DB, it stops with ERROR: role $someDBRole does not exist. This is because the psql v9.2 dump SQL files DROP roles before reinstating them; the container DB does not like this. Unfortunately it is not until psql v9.4 that pg_dumpall and pg_dump have the option to --if-exists (see pg_dumpall v9.2 documentation). What would you suggest that I do in order to remedy this? I could manually edit the SQL dump files, but this would be impractical as the snapshots of the legacy DB need to be automated. Is there a way to suppress this error during container startup?
If I want to convert from ASCII to UTF-8, is it adequate to simply set the encoding option for pg_dumpall and pg_dump? Or do I need to take into consideration other issues when upgrading?
Is there a way to supress the removal and adding of the postgres super user which is in the dump SQL?
In general are there any other gotchas when containerising and/or updating a postgres DB.
I'm not familiar with Docker so I don't know how straightforward it'll be do to these things, but in general, pg_dump/dumpall output, when it's in SQL format, will work just fine after having gone through some ugly string manipulation.
Pipe it through sed -e 's/DROP ROLE/DROP ROLE IF EXISTS/', ideally when writing the .sqls, but it's fine to just run sed -i -e <...> to munge the files in-place after they're created if you don't have a full shell available. Make it sed -r -e '/^DROP ROLE/DROP ROLE IF EXISTS/ if you're worried about strings containing DROP ROLE in your data, at the cost of portability (AFAIK -r is a GNU addition to sed).
Yes. It's worth checking the data in pg12 to make sure it got imported correctly, but in the general case, pg_dump has been aware of encoding considerations since time immemorial, and a dump->load is absolutely the best way to change your DB encoding.
Sure. Find the lines that do it in your .sql, copy enough of it to be unique, and pipe it through grep -v <what you copied> :D
I can't speak to the containerizing aspect of things, but - and this is more of a general practice, not even really PG-specific - if you're dealing with a large DB that's getting migrated, prepare a small one, as similar as possible to the real one but omitting any bulky data, to test with to get everything working so that doing the real migration is just a matter of changing some vars (I guess $REMOtE_HOST and $REMOTE_PORT in your case). If it's not large, then just be comfortable blowing away any pg12 containers that failed partway through the import, figure out & do whatever to fix the failure, and start from the top again until it works end-to-end.
I made a backup of a database of postgres, it is not the first time I do it, I used this command:
pg_dump db -f /backup/agosto_31.sql
And I do the restore with this:
psql -d August_31 -f August_31.sql
But this time I did not import any trigger, and there are many. I checked in the file August_31.sql and they are. How could I import them again? Only the triggers.
Thanks everyone, greetings!
There are not any possibility to import only triggers. But, because your dump (backup file) is in SQL format (plain text), you can cut trigger definition manually in any editor. For this case is practical dump data and schema to separate files.
Is possible so import (restore from backup) fails, probably due broken dependencies. Check psql output if there are some errors. psql has nice possibility to stop on first error:
psql -v ON_ERROR_STOP=1
Use this option. Without error specification is not possible to help more.
I have a site that uses PostgreSQL. All content that I provide in my site is created at a development environment (this happens because it's webcrawler content). The only information created at the production environment is information about the users.
I need to find a good way to update data stored at production. May I restore to production only the tables updated at development environment and PostgreSQL will update this records at production or the best way would be to backup the users information at production, insert them at development and restore the whole database at production?
Thank you
You can use pg_dump to export the data just from the non-user tables in the development environment and pg_restore to bring that into prod.
The -t switch will let you pick specific tables.
pg_dump -d <database_name> -t <table_name>
https://www.postgresql.org/docs/current/static/app-pgdump.html
There are many tips arounds this subject here and here.
I'd suggest you to take a look on these links before everything.
If your data is discarded at each update process then a plain dump will be enough. You can redirect pg_dump output directly to psql connected on production to avoid pg_restore step, something like below:
#Of course you must drop tables to load it again
#so it'll be reasonable to make a full backup before this
pg_dump -Fp -U user -h host_to_dev -T=user your_db | psql -U user -h host_to_production your_db
You might asking yourself "Why he's saying to drop my tables"?
Bulk loading data on a fresh table is faster than deleting old data and inserting again. A quote from the docs:
Creating an index on pre-existing data is quicker than updating it incrementally as each row is loaded.
Ps¹: If you can't connect on both environment at same time then you need to do pg_restore manually.
Ps²: I don't recommend it but you can append --clean option on pg_dump to generate DROP statements automatically. Be extreme careful with this option to avoid dropping unnexpected objects.
How to take an automatic backup of a PostgreSQL database in Ubuntu?
Or is there a script available to take time-to-time PostgreSQL database backups?
you can use the following:
sudo crontab -e
at the end of the file add this:
0 6 * * * sudo pg_dump -U USERNAME -h REMOTE_HOST -p REMOTE_PORT NAME_OF_DB > LOCATION_AND_NAME_OF_BACKUP_FILE
This command will take an automated backup of your selected db everyday at 6:00 AM (after changing options of the command to fit to ur db)
It is advisable to create a backup every time with a new name in order to be able to restore data to a specific date. Also good practice to send notifications in case the backups fail.
Here is a good script for automatic backup, as well as general recommendations for automating backups:
How to Automate PostgreSQL Database Backups
I had to delete from some records from a table so before doing that I took this table's backup using mysqldump command.
After deleting I realized there has been a mistake so I thought to myself...NO PROBLEM....I have the backup so I will restore it.
I thought that when you export then you use DATABASE > FILE.SQL so importing would mean DATABASE < FILE.SQL
So I ran the following command:
mysqldump -u USER -p DATABASE < FILE.SQL
And then I realized I have made a mistake because it started displaying all records on screen. Like it's running in verbose mode. I sat there completely still and frozen thinking as to what the hell is happening!!!! because it was even displaying those tables, which I haven't exported earlier. I didn't know whether to stop it or let it complete the process. I waited for about 2, 3 minutes and then pressed Ctrl + C to stop it and prayed to God that everything is fine. Then I checked my database in phpMyAdmin and apparently everything is fine, all tables are present and so is data.
So my question is.....WHAT DID IT DO WHEN I RAN IT WITH <
This was a live database and I almost shat my pants trust me!
Thanks
Mysqldump does not take any <, so it just re-dump your database into your terminal,
and that's the reason your screen is flood with the data (in short, no problem at all)
You use mysqldump to back up a database and mysql to restore it:
mysql -u USER -p DATABASE < FILE.SQL
I don't think it did anything. mysqldump is a write-only tool and to my knowledge, never accepts command line input. You fed it garbage it didn't understand, no harm done.