We're developing several Lagom-based Scala micro-services. They are configured using variable replacement in application.conf, eg.
mysql = {
url = "jdbc:mysql://"${?ENV_MYSQL_DATABASE_URL}
During development, we set these variables as Java System Properties via a env.sbt file that calls System.setProperty("ENV_MYSQL_DATABASE_URL", url). This is working fine.
Now I want to deploy this in a container to my local Docker installation. We are using the SbtReactiveAppPlugin to build the Docker image from build.sbt and simply run sbt Docker/publishLocal. This works as expected, a Docker image is created and I can fire it up.
However, passing in environment variables using the standard docker or docker-compose mechanisms does not seem to work. While I can see that the environment variables are set correctly inside the Docker container (verified using env on a bash and also by doing log.debug("ENV_MYSQL_DATABASE_URL via env: " + sys.env("ENV_MYSQL_DATABASE_URL")) inside the service), they are not used by the application.conf and not available in the configuration system. The values are empty/unset (verified through configuration.getString("ENV_MYSQL_DATABASE_URL").toString() and the exceptions thrown by the mysql system and other systems).
The only way I've gotten it to work was by fudging this into the JAVA_OPTS via JAVA_OPTS=-D ENV_MYSQL_DATABASE_URL=..... However, this seems like a hack, and doesn't appear to scale very well with dozens of environment parameters.
Am I missing something, is there a way to easily use the environment variables inside the Lagom application and application.conf?
Thanks!
I've used Lightbend config to configure Lagom services via environment variables in docker containers for many years, so know that it can be done and has been pretty straightforward in my experience.
With that in mind, when you say that they're not used by application.conf, do you mean that they're unset? Note that unless you're passing a very specific option as a Java property, configuration.getString("ENV_MYSQL_DATABASE_URL") will not read from an environment variable, so checking that will not tell you anything about whether mysql.url is affected by the environment variable. configuration.getString("mysql.url") will give you a better idea of what's going on.
I suspect that in fact your Docker image is being built with the dev-mode properties hardcoded in, and since Java system properties take precedence over everything else, they're shadowing the environment variable.
You may find it useful to structure your application.conf along these lines:
mysql_database_url = "..." # Some reasonable default default for dev-mode
mysql_database_url = ${?ENV_MYSQL_DATABASE_URL}
mysql {
url = "jdbc://"${mysql_database_url}
}
In this case, you have a reasonable default for a developer (probably including in the docs some instructions for running MySQL in a way compatible with that configuration). The default can then be overridden via setting a Java property (e.g. JAVA_OPTS=-Dmysql_database_url) or by setting the ENV_MYSQL_DATABASE_URL environment variable.
While I agree with the answer provided by Levi Ramsey, I would suggest you to use typesafe's config to load the your config
Related
I am using ASP.NET Core 5.0 and I have a Web API app deployed to internal cloud where few settings like DB are controlled via environment variables on the host cloud. In my Startup.cs I have the below code
string projectDbConnection = Configuration.GetSection("ProjectDatabaseSettings").GetValue<string>("PROJECT_DB_CONNECTION");
string projectDbName = Configuration.GetSection("ProjectDatabaseSettings").GetValue<string>("PROJECT_DB_NAME");
Here as I understand, when running locally in IIS Express it looks for appsettings.<Environment>.json and they take precedence over appsettings.json values.
But this app is always connecting to the wrong DB when I deployed to Cloud where I mentioned the PROJECT_DB_CONNECTION & PROJECT_DB_NAME as Environment variables for the app.
To make the app read from the Environment variables I had to change the above Code in Startup.cs as
string projectDbConnection = Configuration.GetValue<string>("PROJECT_DB_CONNECTION");
string projectDbName = Configuration.GetValue<string>("PROJECT_DB_NAME");
I am unable to understand the difference between the GetSection.GetValue and just GetValue and why I should use Configuration.GetValue() to direct app to read from Env variables.
what am I missing and when should we use what?
Naming of environment variables
There is kind of a naming convention in the environment variables for nested appsettings to env vars, see naming of environment variables.
Each element in the hierarchy is separated by a double underscore.
In your case it would work if you name the env variable: ProjectDatabaseSettings__PROJECT_DB_CONNECTION.
Config Order
Regarding to Microsoft Documentation there is a order in which the config sources are checked.
ChainedConfigurationProvider : Adds an existing IConfiguration as a source. In the default configuration case, adds the host configuration and setting it as the first source for the app configuration.
appsettings.json using the JSON configuration provider.
appsettings.Environment.json using the JSON configuration provider. For example, appsettings.Production.json and appsettings.Development.json.
App secrets when the app runs in the Development environment.
Environment variables using the Environment Variables configuration provider.
Command-line arguments using the Command-line configuration provider.
The usecase
This is useful when you are developing local using appsettings.json, but run in a cluster or cloud in production where it is more convenient to use environment variables (f.e.: in kubernetes environment variables are set via config maps).
Adding Property in Scala Environment Properties
val sysProps = System.getProperties
sysProps.setProperty("current.date.time", LocalDateTime.now().toString())
i'm able to save this property.
I tried accessing this property(current.date.time) in log4j.properties like below
log4j.appender.file.File=C:/Users/vsami/Desktop/Demo_${current.date.time}.log
log4j.appender.file.File=C:/Users/vsami/Desktop/Demo_${env:current.date.time}.log
Log file is getting generated in above location like Demo_.log, Expected :- Demo_2019/11/27T13:21:00.log
Above implementation is not helping me in accessing variable from environment properties and generate log file with expected naming convention.
JVM has properties that can be passed via -D parameter at VM boot. -Dprop=value.
These properties can be read via System.getProperties API call. See docs for more info.
Environment variables are not specified on JVM boot and managed independently from VM by your boot environment (it can be shell, bash etc). You cannot change environment variables in already running VM. These variables can be read via System.getenv()
$ is a look up operator in log4j and can be used to resolve env variables with env: prefix or Main Arguments Lookup with prefix main:.
You could use main:current.date.time and initialise your value as following
MainMapLookup.setMainArguments(Array("--current.date.time", LocalDateTime.now().toString()));
Make sure that MainMapLookup is called before logging is initialised.
I could set http.port in applicaton.conf by using playframework1.2.7
like this
http.port = 9020
jpda.port = 8020
also jdpa.port.
But in play2.4.
I cannot set http.port in application.conf like this.
I know that I can do like this when I run this project.
activator "run 9020"
But it is too troublesome for me.
If you have some ideas,
please share your idea.
You cannot specify port in aaplication.conf during run mode (but this can be used while deploying).
In run mode the HTTP server part of Play starts before the application has been compiled. This means that the HTTP server cannot access the application.conf file when it starts. If you want to override HTTP server settings while using the run command you cannot use the application.conf file. Instead, you need to either use system properties or the devSettings setting shown above.
Source: https://www.playframework.com/documentation/2.4.x/Configuration#HTTP-server-settings-in-application.conf
Also look at full server configuration options
https://www.playframework.com/documentation/2.4.x/ProductionConfiguration#Server-configuration-options
I have REST API, that is developed using Scala and Spray framework. I am able to execute and launch my Api from localhost. The API is connected to the database. The IP Address(localhost) and port of Database is read from the "application.conf" file under the resources.
Everything works fine till I start using Docker. In Docker I have :
1. One Docker container of Rest API
2. One Docker container of Database.
The IP address of Database changes for each docker instance, therefore I need to update my "application.conf" file. Although I can use the hostname of Db instance that remains the same.
My issue is : Can I have two "application.conf" files , one for localhost and one for Docker instance? IS there a way to change the "application.conf" file at the run time.
P.s I am using "sbt run" to run the application and as per documentation it does not support java system properties or environment variables
Yes, you can choose the config at runtime. spray & akka use the typesafe config library which allows setting single settings or the whole configuration using JVM properties.
From the documentation of config:
For applications using application.{conf,json,properties}, system
properties can be used to force a different config source:
config.resource specifies a resource name - not a basename, i.e. application.conf not application
config.file specifies a filesystem path, again it should include the extension, not be a basename
config.url specifies a URL
These system properties specify a replacement for
application.{conf,json,properties}, not an addition. They only
affect apps using the default ConfigFactory.load() configuration. In
the replacement config file, you can use include "application" to
include the original default config file; after the include statement
you could go on to override certain settings.
I have read the documentation concerning falling back to environment variables at https://github.com/typesafehub/config/blob/master/HOCON.md#substitution-fallback-to-environment-variables. My understanding was that it would pickup any envars. So for instance, if from the shell I was able to do echo $HOSTNAME and see a non-empty response then HOCON should do that as well.
In my application.conf I have a line
akka.remote.netty.tcp.hostname = ${HOSTNAME}
However, my app is not happy with this and fails to start with.
/conf/application.conf: 9: Could not resolve substitution to a value: ${HOSTNAME}
Is this a user issue? A shell issue? I am able to login as the user and echo $HOSTNAME
Tagging this scala and akka since that userbase probably has the most exposure to HOCON
The reason for HOCON not picking up the envar is that my app runs as a linux service (Centos 6.5) which clears away most environment variables.
See https://unix.stackexchange.com/questions/44370/how-to-make-unix-service-see-environment-variables for a relevant description of the issue
this is a shot in the dark, but are you using an older version of typesafe-config? maybe its a newer-ish feature? the feature seems to be advertised as you describe, but if you are pulling in typesafe-config as a transient dependency (say from akka), maybe you are getting an older version.
what happens if you remove the substitution in your .conf file (so parsing is successful) and then print out the contents of ConfigFactory.systemEnvironment()? for reference: http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#systemEnvironment--
HOSTNAME isn't an environment variable. It's a bash internal variable. See https://superuser.com/questions/132489/hostname-environment-variable-on-linux for more details.