The invoiceninja/invoiceninja Dockerfile seems to not have any dependency on a database. How/where are the data going to be stored?
it needs running MySQL and storage folder.
read more:
https://github.com/invoiceninja/dockerfiles
Related
I'm working on trying to setup my local database with some mock data to work with. We have a development AWS account with a postgres database. I would like to create a backup of it, export it to my local computer, and restore to my local postgres database.
I've been trying to find how to do this online, but everything I'm finding is on how to backup to AWS and to restore back to AWS. I tried creating a snapshot and exporting it via S3 - but the snapshot doesn't produce a sql file to restore from like I was expecting.
If anyone can point me in the right direction I would very much appreciate it :)
I am afraid that the only chance you have is pg_dump/pg_restore.
Even if Amazon lets you get your hands on its file system backups, which I doubt, they may be of little use to you, since Amazon runs modified versions of PostgreSQL and you cannot be sure that the physical file format is identical to PostgreSQL.
This might be a simple question, but I would like some clarification.
Based off the docs, Heroku has an ephemeral file system. How I interpret it is that anytime you upload a file to Heroku and there is a change in the configuration or the app is restarted, the files are gone.
However, I was wondering if this is the case if you upload data to Heroku Postgres through a dumps file.
For development, I am using a local Postgres server. From there, I would create a dumps file and then upload that file using commands found here:
https://stackoverflow.com/a/71206831/3100570
Now suppose my application makes a POST request to Heroku Postgres, would that data be persisted along with the initial data from the dumps file in the event that the application is restarted or crashed?
Ingesting data into your PostgreSQL database this way doesn't touch your dyno's filesystem. You are simply connecting to PostgreSQL and running the SQL commands contained in that file:
-f, --file=file
SQL file to run
The data will be stored in PostgreSQL in exactly the same way it would if you did a bunch of INSERTs yourself. You should have no problem ingesting data this way and then continuing to interact with your application as normal.
I have an application which, for local development, has multiple Docker containers (organized under Docker Compose). One of those containers is a Postgres 10 instance, based on the official postgres:10 image. That instance has its data directory mounted as a Docker volume, which persists data across container runs. All fine so far.
As part of testing the creation and initialization of the postgres cluster, it is frequently the case that I need to remove the Docker volume that holds the data. (The official postgres image runs cluster init if-and-only-if the data directory is found to be empty at container start.) This is also fine.
However! I now have a situation where in order to test and use a third party Postgres extension, I need to load around 6GB of (entirely static) geocoding lookup data into a database on the cluster, from Postgres backup dump files. It's certainly possible to load the data from a local mount point at container start, and the resulting (very large) tables would persist across container restarts in the volume that holds the entire cluster.
Unfortunately, they won't survive the removal of the docker volume which, again, needs to happen with some frequency. I am looking for a way to speed up or avoid the rebuilding of the single database which holds the geocoding data.
Approaches I have been or currently am considering:
Using a separate Docker volume on the same container to create persistent storage for a separate Postgres tablespace that holds only the geocoder database. This appears to be unworkable because while I can definitely set it up, the official PG docs say that tablespaces and clusters are inextricably linked such that the loss of the rest of the cluster would render the additional tablespace unusable. I would love to be wrong about this, since it seems like the simplest solution.
Creating an entirely separate container running Postgres, which mounts a volume to hold a separate cluster containing only the geocoding data. Presumably I would then need to do something kludgy with foreign data wrappers (or some more arcane postgres admin trickery that I don't know of at this point) to make the data seamlessly accessible from the application code.
So, my question: Does anyone know of a way to persist a single database from a dockerized Postgres cluster, without resorting to a dump and reload strategy?
If you want to speed up then you could convert your database dump to a data directory (import your dump to a clean postgres container, stop it and create a tarball of the data directory, then upload it somewhere). Now when you need to create a new postgres container use use a init script to stop the database, download and unpack your tarball to the data directory and start the database again, this way you skip the whole db restore process.
Note: The data tarball has to match the postgres major version so the container has no problem to start from it.
If you want to speed up things even more then create a custom postgres image with the tarball and init script bundled so everytime it starts then it will wipe the empty cluster and copy your own.
You could even change the entrypoint to use your custom script and load the database data, then call docker-entrypoint.sh so there is no need to delete a possible empty cluster.
This will only work if you are OK with replacing the whole cluster everytime you want to run your tests, else you are stuck with importing the database dump.
My problem is to get big(250Gb) postgres dump on my local machine.
Its on AWS RDS. I tried to dump it to local machine, but it takes too long, kinda 3+ days.
Trying to find a way to dump it into S3 and download from there safely. May be you could suggest more effective way to do that. Will appreciate any kind of help.
Thanks!
As of my knowledge, aws does not provide a way to backup db into s3
you can take a look into this question and answers,
Export huge database from amazon RDS to local mysql
here is one answer
If the data is that big I would suggest copying the RDS snapshot on S3, as explained here.
Link to documentation to copy snapshot to s3
This topic is covered in this StackOverflow thread Exporting a AWS Postgres RDS Table to AWS S3
Another solution would be to spin up an EC2 instance and dump the database to a local EBS volume that is large enough for the following steps. Then chose one of the following:
Compress the DB dump into multiple files and copy to S3 for download. I would use a smart S3 download manager given the size of the database dump.
Export the S3 data using Snowball Export S3 Data. If your Internet connection is not fast enough / reliable enough then Snowball will get you the data.
I have deployed my SpringBoot app to Heroku. Now I would like to copy my local PostgreSQL to Heroku.
I have found some information on devcenter.heroku.com.
However I don't understand enough about the using of file db.changelog-master.yaml.
Could anyone give me details about the simplest solutions to copy the database?
Create a valid dump of your local postgres database and host it somewhere publicly available. Now you will be able to restore this entire dataset (schema and records) with pg:backups:restore as shown here. The sole caveat here is that the target database must be completely empty for this to work. You can empty a Heroku postgres database with heroku pg:reset.
If you cannot take the approach listed above then you can run pg_restore directly from your local instance, provided your local version of Postgres is >= the target version of Postgres. This also applies to creating the dumpfile and is a requirement because pg utilities are not guaranteed to be forward compatible. Documentation for pg_restore is here.