Configurations are applied to an AEM instance based on run modes. How AEM determines the config file to be picked in case of multiple runmodes and multiple configurations? Assuming the below configs are available in an AEM project,
/apps
/myproject
- config
- config.prod
- config.author
- config.active
- config.prod.active
- config.prod.author.active
which configuration gets picked or applied to an AEM instance created with run modes author,nosamplecontent,prod,active?
Is there any defined set of rules(or best practices) related to config creation documented that i can refer while setting up the project
Thanks,
Jai
This article talks about the specifics of how they are applied:
https://helpx.adobe.com/experience-manager/6-4/sites/deploying/using/configuring-osgi.html?cq_ck=1368002864971#ConfigurationDetails
Resolution of multiple Run Modes
For run mode specific configurations, multiple run modes can be
combined. For example, you can create configuration folders in the
following style:
/apps/*/config../
Configurations in such folders will be applied if all run modes match
a run mode defined at startup.
For example, if an instance was started with the run modes
author,dev,emea, configuration nodes in /apps//config.emea,
/apps//config.author.dev/ and /apps//config.author.emea.dev/ will be
applied, while configuration nodes in /apps//config.author.asean/ and
/config/author.dev.emea.noldap/ will not be applied.
If multiple configurations for the same PID are applicable, the
configuration with the highest number of matching run modes is
applied.
For example, if an instance was started with the run modes
author,dev,emea, and both /apps//config.author/ and
/apps//config.emea.author/ define a configuration for
com.day.cq.wcm.core.impl.VersionManagerImpl, the configuration
in/apps/*/config.emea.author/ will be applied.
This rule's granularity is at a PID level. You cannot define some
properties for the same PID in/apps//config.author/ and more specific
ones in /apps//config.emea.author/ for the same PID. The
configuration with the highest number of matching run modes will be
effective for the entier PID.
Here are some best practices:
https://aemmastery.com/runmodes-best-practices-8daa8a2c6582
https://blog.kristianwright.com/2013/08/21/aem-best-practice-osgi-configurations/
I think, when you create a aem instance using java -jar command, you can mention the run mode of aem instance prod, stage, author, publish etc.
This is what mapped to apps/myproject - config.* folder names.
configurations are picked up top to low . i.e. .prod will get applied to all prod instances and superseded by one level prod.active. ... once configuration are created, instances running need to have run modes added to it accordingly.
Configurations in such folders will be applied if all run modes match a run mode defined at startup.
As per your example, if an instance was started with the run modes author,nosamplecontent,prod,active
/apps
/myproject
- config ---> will be applied
- config.prod ---> will be applied
- config.author ----> will be applied
- config.active ----> will be applied
- config.prod.active ----> will be applied
- config.prod.author.active ---> will be applied
Configurations that you provide in these folders generally are for specific PID. If multiple configurations for the same PID are applicable, the configuration with the highest number of matching run modes is applied.
So for e.g. if you had configuration for resourceresolver in config.prod as well as config.prod.active (but not in any other folder) then configuration for the resource resolver will be active from config.prod.active since that matches the higher number of runmodes applied.
Related
I am building a CI/CD pipeline to release SF Stateless Application packages into clusters using parameters for everything. This is to ensure environments (DEV/UAT/PROD) can be scoped with different settings.
For example in a DEV cluster an application package may have an instance count of 3 (in a 10 node cluster)
I have noticed that if an application is in the cluster and running with an instance count (for example) of 3, and I change the deployment parameter to anything else (e.g. 5), the application package will upload and register the type, but will fail on attempting to do a rolling upgrade of the running application.
This also works the other way e.g. if the running app is -1 and you want to reduce the count on next rolling deployment.
Have I missed a setting or config somewhere, is this how it is supposed to be? At present its not lending itself to being something that is easily scaled without downtime.
At its simplest form we just want to be able to change instance counts on application updates, as we have an infrastructure-as-code approach to changes, builds and deployments for full tracking ability.
Thanks in advance
This is a common error when using Default services.
This has been already answered multiple times in these places:
Default service descriptions can not be modified as part of upgrade set EnableDefaultServicesUpgrade to true
https://blogs.msdn.microsoft.com/maheshk/2017/05/24/azure-service-fabric-error-to-allow-it-set-enabledefaultservicesupgrade-to-true/
https://github.com/Microsoft/service-fabric/issues/253#issuecomment-442074878
I'm setting up a distributed cluster for ZooKeeper based on version 3.5.2. In specific, I'm utilizing the reconfig command to dynamically update the configuration when there is any rebalance in the cluster (e.g. one of the nodes comes down).
The observation I have is that the zoo.cfg.dynamic file is not getting updated even when the reconfig (add/remove) command is correctly executed. Is this the expected behavior ? Basically I'm looking for guidance whether we should manage the zoo.cfg.dynamic file also through a separate script (update it lock-step with the reconfig command) or can we rely on the reconfig command to do this for us. My preference/expectation is the latter.
Following is the sample command:
reconfig -remove 6 -add server.5=125.23.63.23:1234:1235;1236
From the reconfig documentation:
Dynamic configuration parameters are stored in a separate file on the server (which we call the dynamic configuration file). This file is linked from the static config file using the new dynamicConfigFile keyword.
So I could practically start with any file name to host the ensemble list and ensure the 'dynamicConfigFile' config keyword just point to this file.
Now when the reconfig command is run, basically a new dynamic-config file (e.g. zoo.cfg.dynamic.00000112) is generated which contains the transformed list of servers, in the form as below (as an example):
server.1=125.23.63.23:2780:2783:participant;2791
server.2=125.23.63.24:2781:2784:participant;2792
server.3=125.23.63.25:2782:2785:participant;2793
The zoo.cfg file is hence auto-updated to point the 'dynamicConfigFile' config keyword to the new config file (zoo.cfg.dynamic.00000112). The previous dynamic-config file continues to be available in the runtime (config directory) but it is not being referred by the main config anymore.
So overall, there is no overhead to update any file lock-step to the reconfig command i.e. reconfig command takes care of it all. The only potential overhead to upfront resolve is to write a periodic purge of the old dynamic-config files.
The One Binary principle explained here:
http://programmer.97things.oreilly.com/wiki/index.php/One_Binary states that one should...
"Build a single binary that you can identify and promote through all the stages in the release pipeline. Hold environment-specific details in the environment. This could mean, for example, keeping them in the component container, in a known file, or in the path."
I see many dev-ops engineers arguably violate this principle by creating one docker image per environment (ie, my-app-qa, my-app-prod and so on). I know that Docker favours immutable infrastructure which implies not changing an image after deployment, therefore not uploading or downloading configuration post deployment. Is there a trade-off between immutable infrastructure and the one binary principle or can they complement each-other? When it comes to separating configuration from code what is the best practice in a Docker world??? Which one of the following approaches should one take...
1) Creating a base binary image and then having a configuration Dockerfile that augments this image by adding environment specific configuration. (i.e my-app -> my-app-prod)
2) Deploying a binary-only docker image to the container and passing in the configuration through environment variables and so on at deploy time.
3) Uploading the configuration after deploying the Docker file to a container
4) Downloading configuration from a configuration management server from the running docker image inside the container.
5) Keeping the configuration in the host environment and making it available to the running Docker instance through a bind mount.
Is there another better approach not mentioned above?
How can one enforce the one binary principle using immutable infrastructure? Can it be done or is there a trade-off? What is the best practice??
I've about 2 years of experience deploying Docker containers now, so I'm going to talk about what I've done and/or know to work.
So, let me first begin by saying that containers should definitely be immutable (I even mark mine as read-only).
Main approaches:
use configuration files by setting a static entrypoint and overriding the configuration file location by overriding the container startup command - that's less flexible, since one would have to commit the change and redeploy in order to enable it; not fit for passwords, secure tokens, etc
use configuration files by overriding their location with an environment variable - again, depends on having the configuration files prepped in advance; ; not fit for passwords, secure tokens, etc
use environment variables - that might need a change in the deployment code, thus lessening the time to get the config change live, since it doesn't need to go through the application build phase (in most cases), deploying such a change might be pretty easy. Here's an example - if deploying a containerised application to Marathon, changing an environment variable could potentially just start a new container from the last used container image (potentially on the same host even), which means that this could be done in mere seconds; not fit for passwords, secure tokens, etc, and especially so in Docker
store the configuration in a k/v store like Consul, make the application aware of that and let it be even dynamically reconfigurable. Great approach for launching features simultaneously - possibly even accross multiple services; if implemented with a solution such as HashiCorp Vault provides secure storage for sensitive information, you could even have ephemeral secrets (an example would be the PostgreSQL secret backend for Vault - https://www.vaultproject.io/docs/secrets/postgresql/index.html)
have an application or script create the configuration files before starting the main application - store the configuration in a k/v store like Consul, use something like consul-template in order to populate the app config; a bit more secure - since you're not carrying everything over through the whole pipeline as code
have an application or script populate the environment variables before starting the main application - an example for that would be envconsul; not fit for sensitive information - someone with access to the Docker API (either through the TCP or UNIX socket) would be able to read those
I've even had a situation in which we were populating variables into AWS' instance user_data and injecting them into container on startup (with a script that modifies containers' json config on startup)
The main things that I'd take into consideration:
what are the variables that I'm exposing and when and where am I getting their values from (could be the CD software, or something else) - for example you could publish the AWS RDS endpoint and credentials to instance's user_data, potentially even EC2 tags with some IAM instance profile magic
how many variables do we have to manage and how often do we change some of them - if we have a handful, we could probably just go with environment variables, or use environment variables for the most commonly changed ones and variables stored in a file for those that we change less often
and how fast do we want to see them changed - if it's a file, it typically takes more time to deploy it to production; if we're using environment variable
s, we can usually deploy those changes much faster
how do we protect some of them - where do we inject them and how - for example Ansible Vault, HashiCorp Vault, keeping them in a separate repo, etc
how do we deploy - that could be a JSON config file sent to an deployment framework endpoint, Ansible, etc
what's the environment that we're having - is it realistic to have something like Consul as a config data store (Consul has 2 different kinds of agents - client and server)
I tend to prefer the most complex case of having them stored in a central place (k/v store, database) and have them changed dynamically, because I've encountered the following cases:
slow deployment pipelines - which makes it really slow to change a config file and have it deployed
having too many environment variables - this could really grow out of hand
having to turn on a feature flag across the whole fleet (consisting of tens of services) at once
an environment in which there is real strive to increase security by better handling sensitive config data
I've probably missed something, but I guess that should be enough of a trigger to think about what would be best for your environment
How I've done it in the past is to incorporate tokenization into the packaging process after a build is executed. These tokens can be managed in an orchestration layer that sits on top to manage your platform tools. So for a given token, there is a matching regex or xpath expression. That token is linked to one or many config files, depending on the relationship that is chosen. Then, when this build is deployed to a container, a platform service (i.e. config mgmt) will poke these tokens with the correct value with respect to its environment. These poke values most likely would be pulled from a vault.
I have a number of legacy services running which read their configuration files from disk and a separate daemon which updates these files as they change in zookeeper (somewhat similar to confd).
For most of these types of configuration we would love to move to a more environment variable like model, where the config is fixed for the lifetime of the pod. We need to keep the outside config files as the source of truth as services are transitioning from the legacy model to kubernetes, however. I'm curious if there is a clean way to do this in kubernetes.
A simplified version of the current model that we are pursuing is:
Create a docker image which has a utility for fetching config files and writing them to disk ones. Then writes a /donepath/done file.
The main image waits until the done file exists. Then allows the normal service startup to progress.
Use an empty dir volume and volume mounts to get the conf from the helper image into the main image.
I keep seeing instances of this problem where I "just" need to get a couple of files into the docker image at startup (to allow per-env/canary/etc variance), and running all of this machinery each time seems like a burden throw on devs. I'm curious if there is a more simplistic way to do this already in kubernetes or on the horizon.
You can use the ADD command in your Dockerfile. It is used as ADD File /path/in/docker. This will allow you to add a complete file quickly to your container. You need to have the file you want to add to the image in the same directory as the Dockerfile when you build the container. You can also add a tar file this way which will be expanded during the build.
Another option is the ENV command in a your Dockerfile. This adds the data as an environment variable.
Scenario: The same configuration parameter is present under /libs/*/config and /apps/*/config and modified in both locations as well as via Felix console.
And during run-time somehow Felix console configuration is taking precedence. How? My understanding is /apps should take precedence. Any views.
You are correct. See
http://docs.adobe.com/docs/en/cq/current/deploying/configuring_osgi.html#Configuration%20Details which states:
The following order of precedence is used:
Repository nodes under /apps/*/config....either with type sling:OsgiConfig or property files (CHECK)
Repository nodes with type sling:OsgiConfig under /libs/*/config.... (ootb defns)
Any .config files from /crx-quickstart/launchpad/config/.... on the local file system.
This means that a generic configuration in /libs can be masked by a
project specific configuration in /apps.
Are the config nodes run-mode specific? If a run-mode doesn't match, that could explain why the node under apps is not getting applied:
As the same configuration parameter can be located in several places,
the system:
searches for all nodes of type sling:OsgiConfig
filters according to service name
filters according to run mode
Also mentioned in the above doc, in case the changes were made when the system is running, then the following order of precedence applies:
Modifying a configuration in the Web console will take immediate effect as it takes precedence at runtime.
Modifying a configuration in /apps will take immediate effect.
Modifying a configuration in /libs will take immediate effect, unless it is masked by a configuration in /apps.
This might explain why the config changes done in the OSGI console is being picked instead of the changes in your /apps.
The following order of precedence is used:
Repository nodes under /apps/*/config....either with type sling:OsgiConfig or property files (CHECK)
Repository nodes with type sling:OsgiConfig under /libs/*/config.... (ootb defns)
Any .config files from /crx-quickstart/launchpad/config/.... on the local file system.
This means that a generic configuration in /libs can be masked by a project specific configuration in /apps.
In CQ5. Any changes done in felix console modifies the config files with the highest priority.
For eg under
/apps/*/config you have an "org.apache.sling.security.impl.ReferrerFilter.config" file
And you modify the Apache Sling Referrer Filter via felix console
The config file "org.apache.sling.security.impl.ReferrerFilter.config" present under /apps/*/config will be modified with the value you just entered in felix console.
Resolution order at Runtime
Configuration changes made while the system is running trigger a reload with the modified configuration.
Then the following order of precedence applies:
Modifying a configuration in the Web Console will take immediate effect as it takes precedence at runtime.
Modifying a configuration in /apps will make immediate effect.
Modifying a configuration in /libs will take immediate effect, unless it is masked by a configuration in /apps.
Resolution order at startup
Repository nodes under /apps/*/config….. either with type sling:OsgiConfig or property files
Repository nodes with type sling:OsgiConfig under /libs/*/config…. (ootb)
Any .config files from /crx-quickstart/launchpad/config/… on the local file system