How to include MySQL database schema on GitHub? - github

Stackoverflow and MySQL-via-command-line n00b here, please be gentle! I've been looking around for answers to my question but could only find topics dealing with GitHubbing MySQL dumps (as in: data dumps) for collaboration or MySQL "version control" via GitHub, neither of which tells me what I want to know:
How does one include MySQL database schemas/information on tables with PHP projects on GitHub?
I want to share a PHP project on GitHub which relies on the existence of a MySQL database with certain tables. If someone wanted to copy/make use of this project, they would need to have these particular tables in place to make the script work (all tables but one are empty in the beginning and only get filled by the user over time, via the script; the non-empty table holds three values from the start). How does one go about this, what is common practice?
Would I just get a (complete) dump file of my own db/tables, then
delete all the data parts (except for that one non-empty
table), set all autoincrements to zero and then upload that .sql file
to GitHub along with the rest of the project?
OR
Is it best/better practice to write a (PHP) script with which the
(maybe not-so-experienced) user can create these tables without
having to use mysqldump/command line magic?
If solution #1 is the way to go, would I include further instructions on how to use such a .sql file?
Sorry if my questions sound silly, but as I said above, I myself am new to using the command line for MySQL-related things and had only ever used phpMyAdmin until yesterday (when I created my very first dump file with mysqldump - yay!).

Common practice is to include an install script that creates the necessary tables, so solution #2 would be the way to go.
[edit] That script could ofc just replay a dump. ;)
You might also be interested in migrations: How to automate migration (schema and data) for PHP/MySQL application

If you want also track database schema changes
You can use git hooks.
In directory [your_project_dir]/.git/hooks add / edit script pre-commit
#!/bin/sh -e
set -o errexit
# -- you can omit next line if not using version table
version=`git log --tags --no-walk --pretty="format:%d" | sed 1q | sed 's/[()]//g' | sed s/,[^,]*$// | sed 's ...... '`
BASEDIR=$(dirname "$0")
# -- set directorey wher schema dump is placed
dumpfile=`realpath "$BASEDIR/../../install/database.sql"`
echo "Dumping database to file: $dumpfile"
# -- dump database schema
mysqldump -u[user] -p[password] --port=[port] [database-name] --protocol=TCP --no-data=true --skip-opt --skip-comments --routines | \
sed -e 's/DEFINER[ ]*=[ ]*[^*]*\*/\*/' > "$dumpfile"
# -- dump versions table and update core vorsiom according to last git tag
mysqldump -u[user] -p[password] --port=[port] [database-name] [versions-table-name] --protocol=TCP --no- data=false --skip-opt --skip-comments --no-create-info | \
sed -e 's/DEFINER[ ]*=[ ]*[^*]*\*/\*/' | \
sed -e "/INSERT INTO \`versions\` VALUES ('core'/c\\INSERT INTO \`versions\` VALUES ('core','$version');" >> "$dumpfile"
git add "$dumpfile"
# --- Finished
exit 0
Change [user], [password], [port], [database-name], [versions-table-name]
This script is executed autamatically by git on each commit. If commiting tag new version is saved to table dump by tag name. If no changes in database, nothing is commited. Make sure if script is executable :)
Your install script can take sql queries from this dump and developer can easy track database changes.

Related

How to restore one database from a .sql file in which there are two databases?

I was sent a .sql file in which there are two databases. Previously, I only dealt with .sql files in which there is one database. I also can't ask to send databases in different files.
Earlier I used this command:
psql -d first_db < /Users/colibri/Desktop/first_db.sql
Databases on the server and locally have different names.
Tell me, please, how can I now restore a specific database from a file in which there are several?
You have two choices:
Use an editor to delete everything except the database you want from the SQL file.
Restore the whole file and then drop the database you don't need.
The file was probably generated with pg_dumpall. Use pg_dump to dump a single database.
If this is the output of pg_dumpall and the file is too big to edit with something like vi, you can use a stream editor to isolate just what you want.
perl -ne 'print if /^\\connect foobar/.../^\\connect/' < old.sql > new.sql
The last dozen or so lines that this captures will be setting up for and creating the next database it wants to restore, so you might need to tinker with this a bit to get rid of those if you don't want it to attempt to create that database while you replay. You could change the ending landmark to something like the below so that it ends earlier, but that is more likely to hit false positives (where the data itself contains the magic string) than the '^\connect' landmark is.
perl -ne 'print if /^\\connect foobar/.../^-- PostgreSQL database dump complete/'

What is the purpose of the sql script file in a tar dump?

In a tar dump
$ tar -tf dvdrental.tar
toc.dat
2163.dat
...
2189.dat
restore.sql
After extraction
$ file *
2163.dat: ASCII text
...
2189.dat: ASCII text
restore.sql: ASCII text, with very long lines
toc.dat: PostgreSQL custom database dump - v1.12-0
What is the purpose of restore.sql?
toc.dat is binary, but I can open it and it looks like a sql
script too. How different are between the purposes of restore.sql
and toc.dat?
The following quote from the document does't answer my question:
with one file for each table and blob being dumped, plus a so-called Table of Contents file describing the dumped objects
in a machine-readable format that pg_restore can read.
Since a tar dump contains restore.sql besides the .dat files,
what is the difference between the sql script files restore.sql and toc.dat in a tar dump and a
plain dump (which has only one sql script file)?
Thanks.
restore.sql is not used by pg_restore. See this comment from src/bin/pg_dump/pg_backup_tar.c:
* The tar format also includes a 'restore.sql' script which is there for
* the benefit of humans. This script is never used by pg_restore.
toc.dat is the table of contents. It contains commands to create and drop each object in the dump and is used by pg_restore to create the objects. It also contains COPY statements that load the data from the *.dat file.
You can extract the table of contents in human-readable form with pg_restore -l, and you can edit the result to restore only specific objects with pg_restore -L.
The <number>.dat files are the files containing the table data, they are used by the COPY statements in toc.dat and restore.sql.
This looks a script to restore the data to PostgresQL. the script was created using pg_dump.
If you'd like to restore, please have a look at pg_restore.
The dat files contain the data to be restored in those \copy commands in the sql script.
the toc.dat file is not referenced inside the sql file. if you try to peek inside using cat toc.dat|strings you'll find that it contains data very similar to the sql file, but with a few more internal ids.
I think it might have been intended to work without the SQL at some point, but that's not how it's working right now. see the code to generate toc here.

Using triggers with different users (MySQL error 1449)

Question: How can I export the database contents (including triggers) from my dev system and import the data to the live server without running into error 1449 when a trigger gets triggered?
In a recent php project I am making extensive use of mysql triggers but ran into a problem when deploying my database from dev to live system.
E.g. one of my triggers is defined as follows (output generated by using mysqldump)
DELIMITER ;;
/*!50003 CREATE*/ /*!50017 DEFINER=`root`#`localhost`*/ /*!50003 TRIGGER update_template BEFORE UPDATE ON template
FOR EACH ROW BEGIN
SET new.mod_date := now();
END */;;
DELIMITER ;
That trigger was defined on my dev system using the user root#localhost which creates the DEFINER=root#localhost clause in above statement.
root#localhost does not exist as a user on the live server which causes the following error when ever the trigger gets triggered (e.g. by using update templates set...) by the live systems user
1449: The user specified as a definer('root'#'localhost') does not exist
Currently I use mysqldump --add-drop-table --user=root -p my_project > export.sql for export and mysql -u devuser -p my_project < export.sql for importing data.
Export/import works flawless. The error occurs only in cases when I manipulate the data via sql and a trigger gets involved.
Edit:
MySQL version is 5.5.47 (live and dev)
Once you've exported to export.sql, next you'll need to sanitize your trigger lines using regex via sed, like this:
sed -i -- 's/^..!50003\sCREATE.....!50017\sDEFINER=.root...[^]*.....!50003\s\([^;]*\)/CREATE DEFINER=CURRENT_USER \1/g;s/^\s*\([^\*]*;\{0,1\}\)\s\{0,1\}\*\/;;$/\1;;/g' export.sql
This works on Linux, but if you're on Mac OS X, the built-in sed command won't work. You'll need to first brew install gnu-sed, then use gsed instead of sed in the above command.
In my case, the trigger that caused the problem didn't have the BEGIN and END statements. So I applied the corresponding DROP TRIGGER and CREATE TRIGGER, after that I made again a backup that latter restored without problems. i.e:
DROP TRIGGER `incorrect_trg1`;
DELIMITER ;;
CREATE DEFINER = `root`#`localhost` TRIGGER `incorrect_trg1` BEFORE INSERT ON `table1` FOR EACH ROW
BEGIN
SET NEW.col = DATE_FORMAT(NEW.col,'%Y%m');
END;;
DELIMITER ;
Use the following sed command to remove the DEFINER part from the dump file and then import it to your live server.
sed 's/\sDEFINER=`[^`]*`#`[^`]*`//' -i dumpfile.sql
The triggers will then be created by the user importing the dump file by default.

Restore deleted file directly from Eclipse local history

Some git mistakes happened and I lost a lot of changes for one file. I was using Eclipse as my IDE but the git mishap included deleting the project and re cloning the directory. So I can't do the restore from within Eclipse. I believe I have found the local history file that contains the code I want to restore but I'm not sure how to cat this file. It kinda looks like a json.
Anyone know how to restore or read the .metadata.plugins\org.eclipse.core.resources.history
files?
I was able to recover my code.
I went to prj/.metadata/.plugins/org.eclipse.core.resources/.history
Then did some bashing:
fgrep -r -c "[Some function name specific to that file]" * | grep -v ":0" | cut -d : -f 1 | xargs ls -l | grep "Jul 29"
So this is greping and counting the number of times some text specific to the code I'm missing shows up in the files, then removes the files where the count is 0, then removes the count from the end of the file name, then does ls to get details on the files, then optionally search for a specific day. Since I was working with clojure I noticed that there were files that either had a large or small file size. The large files were backups of the REPL, the small files were backups of the code.
+1 for Eclipse :)

How to import Zipped file into Postgres Table

I would like to important a file into my Postgresql system(specificly RedShift). I have found a arguement for copy that allows importing a gzip file. But the provider for the data I am trying to include in my system only produces the data in a .zip. Any built in postgres commands for opening a .zip?
From within Postgres:
COPY table_name FROM PROGRAM 'unzip -p input.csv.zip' DELIMITER ',';
From the man page for unzip -p:
-p extract files to pipe (stdout). Nothing but the file data is sent to stdout, and the files are always extracted in binary
format, just as they are stored (no conversions).
Can you just do something like
unzip -c myfile.zip | gzip myfile.gz
Easy enough to automate if you have enough files.
This might only work when loading redshift from S3, but you can actually just include a "gzip" flag when copying data to redshift tables, as described here:
This is the format that works for me if my s3 bucket contains a gzipped .csv.
copy <table> from 's3://mybucket/<foldername> '<aws-auth-args>' delimiter ',' gzip;
unzip -c /path/to/.zip | psql -U user
The 'user' must be have super user right else you will get a
ERROR: must be superuser to COPY to or from a file
To learn more about this see
https://www.postgresql.org/docs/8.0/static/backup.html
Basically this command is used in handling large databases