We're looking into replacing our update system with docker, but we have a unique constraint where all upgrades need to happen offline. The use case is very similar to how you would update router firmware or something from a LAN not connected to the internet.
Currently our users download a patch file, which they then upload to the web interface of our system over a private LAN. Our system applies the patch. It is all implemented with the diff and patch commands. We do the diffing because our codebase is pretty giant but relatively few files change from version to version.
We think switching to docker can help us tremendously for our development, but for production and our update system we need to make sure we can do offline, diff-based updates.
My question boils down to this: are there docker analogs to the diff and patch commands that can be used to update a container offline?
I know docker has commands like docker diff, but as per the documentation it just shows a list of files that have been added, removed or changed from a container. docker save and docker export look like they come close though, but they provide full images whereas I'm after a diff. Similarly there seems to be no way as far as I can tell to use docker load to load a diff.
Thanks!
IMHO you should use private registry for this. Docker image can contain many layers so your update will be just small layer users need to download. Also updating image on local system doesn't affect running containers, so your users will just download new layers on update and then restart running containers from new image. This will be almost same as patch system. So in short from user side it will look like this:
docker pull new layers from your private registry instance (need to be online)
docker stop old container (offline)
docker run new container from updated image
All config files should be stored outside image for example in a docker volume.
Related
I'm running the "apache/sling" Docker image in an Azure App Service. AFAICT, App Service only supports visibility/persistence in the "/home" directory in the container. Therefore, I think I need to change Sling's "sling.home" property from its default "/opt/sling" to something like "/home/sling".
I suspect I have to create my own Docker image based on "apache/sling", but still, between all the possible ways to set "sling.home", in OSGi framework properties, JAR files, command line arguments, etc., I'm lost to figure out what to actually change in my own Docker image. Should I build some part of this from modified source? Or what?
BTW, this is all in order to start working with Sling until my org eventually gets AEM.
I am following this doc.
There is a show stopper in this.
It's explained that, point network bootstrapper jar to _node.conf files in the directory to generate network-related files, but never mentioned how does _node.conf file must look like, especially with configuration for different aspects like H2 DB, PostgresDB etc.
Share a sample link( not cordapp, I just need network setup examples ) or steps to set up a network with h2 & Postgres.
Not many articles available on the internet, everything takes me to the official document and it just increases confusion or convinces me to stop working.
If you run gradle deployNodes, it will generate all the nodes configuration files that are needed by the Network Bootstrapper inside a /build/nodes folder, which will be created by that gradle task.
You can also find all the files in this repository used in an older bootcamp (but still valid) about how to deploy the nodes using Docker. The configuration files are the same.
You can also find an example of node configuration file in the documentation here.
There is also a very good tutorial video on YouTube made by R3 specifically for network bootstrapper.
I'm getting my way through the HLF 2.0 docs and would love to discuss and try out the new features "External Builders and Launchers" and "Chaincode as an external service".
My goal is to run HLF2.0 on an K8s cluster (OpenShift). Does anyone wants to get in touch or has anyone already figured his way through?
Cheers from Germany
Also trying to use the ExternalBuilder. Setup core.yaml, rebuilt the containers to use it. I get an error that on "peer lifecycle chaincode install .tgz...", that the path to the scripts in core.yaml can not be found.
I've added volume bind commands in the peer-base.yaml, and in docker-compose-cli.yaml, and am using the first-network setup. Dropped out the part of the byfn.sh that would connect to the cli container, so that I do that part manually, do the create, join, update anchors successfully, and then try to do the install and fail. However, on the install, I'm failing on the /bin/detect, because it can't find that file to fork/exec it. To get that far, peer was able to read my external configuration, and read the core.yaml file. At the moment, trying the "mode: dev" in the core.yaml which seems to indicate that the scripts and the chaincode will be run "locally", which I think means it should run in the cli container. Otherwise, tried to walk the code to see how the docker containers are being created dynamically, and from what image, but haven't been able to nail that down yet.
As a Vagrant user, when trying Docker I noticed one significant difference between development workflow with Vagrant and with Docker - with Docker I need to rebuild my image every time from scratch, even if I made minor changes in code.
This is major problem for me, because the process of image rebuilding oftenly very redundant and time consuming.
Perhaps there are some smart workflows with Docker already invented, if so, what are they?
I filed a feature-request for the vagrant-cachier plugin for saving docker build data and attached a bash workaround for that process. If it's okay for you to hack yourself around you can implement the scripts in vagrant.
caching docker build data with vagrant
Note that this procedure needs the vagrant-cachier plugin to be installed and has to save and load +300MB files from disk if they are new to the machine. Thus it's really slow if you have dockerfiles with just 1-5 lines of code but it's fast if you have dockerfiles with a lot of LOCs or images that have to be downloaded from the net.
Also note that this approach saves every intermediate building step. So if you are building an image and change a line in the middle of a dockerfile and build again the docker build process will get all cached intermediate containers till the changed line.
Using baseimages is still the preferred way but you can combine both procedures.
Feel free to post improvements and subscribe so fgrehm will maybe implement this in his plugin natively.
As Mark O'Connor suggested, one of the tips may be building a base image to your container(s). This image should have the dependencies, package installation, downloads... or any other consuming activity. This base image should be supposed to be built less frequently than the other one(s). In a similar way, if the final states of the execution of each step of your dockerfile doesn't change, Docker don't build this layer again. Thus, you can trying execute the commands than may change this state almost every run (e.g.: apt-get update) as later as you can, so docker don't have to rebuild the steps before. And also you can try to edit your dockerfiles in the later steps better than in the first.
Another option if you compile/download something inside the container is to have it downloaded or compiled in a host folder, and attach it to the container using -v or --volume option in docker run.
Finally there is other approaches to this issue as the one used by chef with knife container. In this approach you build the container using chef cookbooks, and each time you build it (because you have edited your cookbooks...) these changes are applied as a new docker layer (AUFS layer) and you don't have to repeat all the process. I didn't recommend this solution unless you have experience with Chef and you have cookbooks to manage your software. You should work harder to get it working and if you want Chef only to manage docker containers I think it doesn't worth it (although chef is a great option to manage infrastructures).
To automate the building process in case you have several images dependents itself, you can have a bash script that helps you with that task (credits to smola#github):
#!/bin/bash
IMAGES="${IMAGES:-stratio/base:test stratio/mesos:test stratio/spark-mesos:test stratio/ingestion:test}"
LATEST_TAG="${LATEST_TAG:-test}"
for image in $IMAGES ; do
USER=${image/\/*/}
aux=${image/*\//}
NAME=${aux/:*/}
TAG=${aux/*:/}
DIR=${NAME}/${TAG}
pushd $DIR
docker build --tag=${USER}/${NAME}:${TAG} .
if [[ $TAG = $LATEST_TAG ]] ; then
docker tag ${USER}/${NAME}:${TAG} ${USER}/${NAME}:latest
fi
popd
done
There are a couple of tricks that might better your workflow (very web-focused)
Docker caching
Always make sure you are adding your source to your Docker image in Dockerfile at the very end.
Example;
COPY data/package.json /data/
RUN cd /data && npm install
COPY data/ /data
This will make sure you get optimal caching when building the image, and that Docker doesn't have to rebuild the npm packages when you are changing your source.
Also, make sure you don't have a base-image that adds folders/files that are often changed (like base images doing COPY . /data/
fig mount
Use fig (or another tool), and mount your source directory when developing. This way, you can develop with instant changes and still use the current version of your code when building the image.
development server
You can start your developer web-server when you are developing, and nginx when not (if you are developing an www app, but same idea applies to other apps).
Example, in your startup script, do something like:
if [[ $DEBUG ]]; then
/usr/bin/supervisorctl start gulp
else
/usr/bin/supervisorctl start nginx
fi
And have autostart=false in your supervisord.conf files.
auto-refresh app
If you are developing a web-app, use tools like gulp and eg gulp-connect, if you are developing a python/django app, use the runserver utility. Both reloads the server when detecting changes in the files.
If you are using the if [[ $DEBUG ]] ... trick, make them listen on the same port as your normal instance (nginx). That way, you can have 1 configuration for your reverse proxy, ie, just send the traffic to example www:8080, it will hit your web-page both in production and if you are developing.
Create a based image that holds the bulk of your application's dependencies. This will significantly reduce your docker build times.
Summary:
I created a patch in my local machine, but I need to apply this patch in a remote machine.
I'm using Tortoise in both machines. How can I do this in a proper way ?
Context:
I have development environment in a project that is not very common, I guess. I could develop it in a remote VM, but the Eclipse and the entire machine are so slow that I think is unproductive. Also, I have to use a VPN to connect to the VM, which makes my connection slower. Because of that, I want to develop in my local machine, but, for the build I need to apply this patch in this remote VM to test if the changes were made correctly. I noticed that a patch can't be applied to a unversioned file, for that, I have to clean my entire remote repository with Tortoise and apply the patch again. But I wonder if this is the best approach.
If you are working with two build environments, you have to version ALL of the source files. The only thing that will not be under version control are the build directory and the machine-specific configuration files.
So if a file is not under version control somewhere, it is likely because your project setup is not correct. Take the time put everything in a single folder that can be under version control, start tracking it and then have the two machines communicating with the same repository.
Side note : it is quite common to develop on a machine and build on an other, you should be able to get a simple and efficient work environment quite easily.
I hope I got you question right. If not, please provide more specific info like your project's tree, the reason why you cannot test on your development machine, why is this specific file not under version control and anything else relevant.