I have database migrations (with sqlalchemy-migrate) working well in my dev environment. However, I'm a little stumped about how to integrate this into my deployment process.
I'd like to use fabric to execute the manage.py file on the remote server but I'm not sure what to use for the repository value in this file. Referring to both 'appname/migrations' and '/usr/local/pylons/appname/env/lib/python2.6/site-packages/appname-05.egg/appname/migrations/'
both fail with a migrate.versioning.exceptions.InvalidRepositoryError
Does anyone have a fabfile and manage.py that plays nicely with sqlalchemy-migrate?
What I did was to generate a manage.py file per the sqlalchemy-migrations docs. In there I hacked it up to load our config information which includes the db auth info. In our case it's a Pylons app so it reads the proper Pylons config.ini file.
http://readthedocs.org/docs/sqlalchemy-migrate/en/latest/versioning.html#project-management-script
Then the fabric commands all interact with the manage.py file vs using the Python API directly. Since everything, from the SA-Migrate manage.py through the app itself I don't run into any sort of path issues like you mention.
Not sure this is a 'exact' fix but maybe helps.
Related
I'm new to deploying an app, as this is the first time I've done it. I'm trying to deploy my mern app to an aws ec2 instance with the intention to use mongodb atlas, I'm using a ubuntu 20.04 server.
I'm following this guide https://keithweaverca.medium.com/setting-up-mern-stack-on-aws-ec2-6dc599be4737 and I've got to part after I've pulled my github repo to the ubuntu server, everything is working up to yet, I skipped the mongodb section of the guide, as I have already created a mongodb cluster.
I'm not sure what to do now, my repo folder is split like this...
- frontend
- build
- public
- src
- Components
- images
... other files in src
- package.json
- backend
- server.js
- models
- routes
- package.json
... a few other irrelevant files (node_modules, etc.)
Although the guide doesn't say to, don't I need to run 'npm run build' in the frontend directory? after I've ran 'npm install'?
If so, my code isn't compiling and I'm not sure why, it says there is a memory leak, and javascript overload.
I know this is a lot of questions, without much detail, but I don't even understand this well enough to give more info. Any questions please ask. I just need to know what commands to run after I've pulled my code from the github repo.
I'm using docker-compose on a Rails project, with the main web service being called web.
When I try to run a test from RubyMine, it attempts to run
/usr/local/bin/docker-compose -f
/Users/jy/Development/#Rails/project/docker-compose.yml -f
/Users/jy/Library/Caches/RubyMine2018.3/tmp/docker-compose.override.35.yml
up --exit-code-from web --abort-on-container-exit web
Even though the web container is already up.
This leads to issues with duplicate networks being created, and the web service being stopped afterward thanks to the --abort-on-container-exit.
How can I make RubyMine run my tests using a simple docker-compose exec web bundle exec rspec …, without all the preamble? I know that command works because it works from the command line (but running an individual test involves a lot of typing to fill in --example testname!)
Apparently, it doesn't support this yet.
https://youtrack.jetbrains.com/issue/RUBY-19849 is the issue that needs resolving in order to make it work properly.
As a Vagrant user, when trying Docker I noticed one significant difference between development workflow with Vagrant and with Docker - with Docker I need to rebuild my image every time from scratch, even if I made minor changes in code.
This is major problem for me, because the process of image rebuilding oftenly very redundant and time consuming.
Perhaps there are some smart workflows with Docker already invented, if so, what are they?
I filed a feature-request for the vagrant-cachier plugin for saving docker build data and attached a bash workaround for that process. If it's okay for you to hack yourself around you can implement the scripts in vagrant.
caching docker build data with vagrant
Note that this procedure needs the vagrant-cachier plugin to be installed and has to save and load +300MB files from disk if they are new to the machine. Thus it's really slow if you have dockerfiles with just 1-5 lines of code but it's fast if you have dockerfiles with a lot of LOCs or images that have to be downloaded from the net.
Also note that this approach saves every intermediate building step. So if you are building an image and change a line in the middle of a dockerfile and build again the docker build process will get all cached intermediate containers till the changed line.
Using baseimages is still the preferred way but you can combine both procedures.
Feel free to post improvements and subscribe so fgrehm will maybe implement this in his plugin natively.
As Mark O'Connor suggested, one of the tips may be building a base image to your container(s). This image should have the dependencies, package installation, downloads... or any other consuming activity. This base image should be supposed to be built less frequently than the other one(s). In a similar way, if the final states of the execution of each step of your dockerfile doesn't change, Docker don't build this layer again. Thus, you can trying execute the commands than may change this state almost every run (e.g.: apt-get update) as later as you can, so docker don't have to rebuild the steps before. And also you can try to edit your dockerfiles in the later steps better than in the first.
Another option if you compile/download something inside the container is to have it downloaded or compiled in a host folder, and attach it to the container using -v or --volume option in docker run.
Finally there is other approaches to this issue as the one used by chef with knife container. In this approach you build the container using chef cookbooks, and each time you build it (because you have edited your cookbooks...) these changes are applied as a new docker layer (AUFS layer) and you don't have to repeat all the process. I didn't recommend this solution unless you have experience with Chef and you have cookbooks to manage your software. You should work harder to get it working and if you want Chef only to manage docker containers I think it doesn't worth it (although chef is a great option to manage infrastructures).
To automate the building process in case you have several images dependents itself, you can have a bash script that helps you with that task (credits to smola#github):
#!/bin/bash
IMAGES="${IMAGES:-stratio/base:test stratio/mesos:test stratio/spark-mesos:test stratio/ingestion:test}"
LATEST_TAG="${LATEST_TAG:-test}"
for image in $IMAGES ; do
USER=${image/\/*/}
aux=${image/*\//}
NAME=${aux/:*/}
TAG=${aux/*:/}
DIR=${NAME}/${TAG}
pushd $DIR
docker build --tag=${USER}/${NAME}:${TAG} .
if [[ $TAG = $LATEST_TAG ]] ; then
docker tag ${USER}/${NAME}:${TAG} ${USER}/${NAME}:latest
fi
popd
done
There are a couple of tricks that might better your workflow (very web-focused)
Docker caching
Always make sure you are adding your source to your Docker image in Dockerfile at the very end.
Example;
COPY data/package.json /data/
RUN cd /data && npm install
COPY data/ /data
This will make sure you get optimal caching when building the image, and that Docker doesn't have to rebuild the npm packages when you are changing your source.
Also, make sure you don't have a base-image that adds folders/files that are often changed (like base images doing COPY . /data/
fig mount
Use fig (or another tool), and mount your source directory when developing. This way, you can develop with instant changes and still use the current version of your code when building the image.
development server
You can start your developer web-server when you are developing, and nginx when not (if you are developing an www app, but same idea applies to other apps).
Example, in your startup script, do something like:
if [[ $DEBUG ]]; then
/usr/bin/supervisorctl start gulp
else
/usr/bin/supervisorctl start nginx
fi
And have autostart=false in your supervisord.conf files.
auto-refresh app
If you are developing a web-app, use tools like gulp and eg gulp-connect, if you are developing a python/django app, use the runserver utility. Both reloads the server when detecting changes in the files.
If you are using the if [[ $DEBUG ]] ... trick, make them listen on the same port as your normal instance (nginx). That way, you can have 1 configuration for your reverse proxy, ie, just send the traffic to example www:8080, it will hit your web-page both in production and if you are developing.
Create a based image that holds the bulk of your application's dependencies. This will significantly reduce your docker build times.
In particular, for test-cases, I want to keep the test database separate so that the test cases don't interfere with development or production databases.
What are some good practices for separating development, test and production environments?
EDIT1: Some context
In Ruby On Rails, there are different configuration files by convention for different environments. So does Play! 2 also support that ?
Or, do I have to cook the configuration files, and then write some glue code that selects the appropriate configuration files ?
At the moment if I run sbt test it uses development database ( configured as "default" in conf/application.conf ). However I would like Play!2 to use a different test database.
EDIT2: On commands that play provides
For Play! 2 framework, I observed this.
$ help play
Welcome to Play 2.2.2!
These commands are available:
-----------------------------
...OUTPUT SKIPPED...
run <port> Run the current application in DEV mode.
test Run Junit tests and/or Specs from the command line
start <port> Start the current application in another JVM in PROD mode.
...OUTPUT SKIPPED...
There are three well defined commands for "test", "development" and "production" instances which are:
test: This runs the test cases. So it should automatically select test configuration.
run <port>: this runs the development instance on the specified port. So this command should automatically select development configuration.
start <port>: this runs the production instance on the specified port. So this should automatically select production configuration.
However, all these commands select the values that are provided in conf/application.conf. I feel there is some gap to be filled here.
Please do correct me if I am wrong.
EDIT3: Best approach is using Global.scala
Described here: How to manage application.conf in several environments with play 2.0?
Good practice is keeping separate instances of the application in separate folders and synching them i.e. via git repo.
When you want to keep single instance you can use alternative configuration file for each environment.
In your application.conf file there is an entry (or entries) for your database, e.g. db.default.url=jdbc:mysql://127.0.0.1:3306/devdb
The conf file can read environment variables using ${?ENV_VAR_NAME} syntax, so change that to something like db.default.url=${?DB_URL} and use environment variables.
A simpler way to get this done and manage your configuration easier is via GlobalSettings. There is a method you can override and that its name is "onLoadConfig". Try check its api at API_LINK
Basically on your conf/ project folder, you'll setup similar to below:
conf/application.conf --> configurations common for all environment
conf/dev/application.conf --> configurations for development environment
conf/test/application.conf --> configurations for testing environment
conf/prod/application.conf --> configurations for production environment
So with this, your application knows which configuration to run for your specific environment mode. For a code snippet of my implementation on onLoadConfig try check my article at my blog post
I hope this is helpful.
I am in the proces of creating a web application in symfony2. I have done all the development locally(OSX), but now I want to show a first version in a presentation where I can not use my own laptop.
I am deploying to an ubuntu 10 server. I also tested it on another small ubuntu 11 vps, but got the same problems.
I double checked with this tutorial whether php is set up correctly for symfony on my ubuntu server:
http://www.joelverhagen.com/blog/2011/05/how-to-configure-symfony-2-0-on-ubuntu-server-2011-4/
For deployment I followed these steps:
Moving app to production mode in Symfony 2
However the first step ( php app/console cache:clear --env=prod --no-debug ) gives me following error (this error also occurs when running this command locally):
php app/console cache:clear --env=prod --no-debug
Clearing the cache for the prod environment with debug false
[ReflectionException]
Class Packed\Bundle\UserBundle\Entity\Rapport does not exist
cache:clear [--no-warmup] [--no-optional-warmers]
Which is a bit weird since Packed\Bundle\UserBundle\Entity\Rapport does not exist, Packed\Bundle\ScoreModelBundle\Entity\Rapport however does.
since the app doe not need to go into production for some time this is not very urgent, what is very urgent is to get it running in development mode, so I tried using it through app_dev.php.
If I remove the localhost-check in app_dev.php everything works as expected but the i8n is not working... I can not switch to other locales as was possible locally:
xxx.com/app_dev.php/en
And it does not use the messages.nl.yml, messages.en.yml, ... and other files.
Any idea what might be causing this?
I checked the symfony and apache log but nothing weird there. Any other logs I should check?
Thanks!
Joris
Try to rebuild your bootstrap file. If you're using APC-cache restart it from cli. If it doesn't help.
Easiest way to do it is run composer and do the update.
To check the translations (the templates should be cached on the server) check the config.yml translator service must be enabled there. You may have it enabled in config_dev.yml but not in the config_prod.yml