My project have multiple spark applications (each with its own spark session) and all of them refers to same log4j.properties file. I want to specify different log4j.properties file (log4j_1.properties) for some of the jobs. I tried below but it is still taking configuration from original file (log4j.properties) only.
Not sure what I am missing here?
conf.set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=file:log4j_1.properties")
Note:
I can see above option set on SparkUI environment tab
Both log4j_1.properties and log4j.properties are there in classpath
Try
-Dlog4j.configuration=log4j_1.properties
Related
I created a shadow jar which included Kafka-client library. But this jar fails to work when placed in $cassaandra_home/conf/trigger directory. Is there a way to add the external jar of Kafka separately and then link it to the main jar?
It might have to be added to Cassandra's CLASSPATH environment variable. You can do this by adding a line in the cassandra-env.sh file, referencing its location:
CLASSPATH="$CLASSPATH:$CASSANDRA_HOME/lib/cassandra-ldap-3.11.4.jar"
The above line allows the use of Instaclustr's Cassandra LDAP Authenticator, referencing it from within Cassandra's lib/ dir (on each node). Give that a try.
I come from the C# .NET world and as I build my Scala project I noticed how many different ways there are to do things in the open source community. Anyways in .Net C# project we have the Web.config or the App.config file which is very useful to specify app keys that are used globally and it is also helpful for transformation for different environments and such.
What would be the equivalent for a Scala SBT project? Do you know a place to store your connection strings and other app settings?
There is no universal configuration file name. The basic way is to use Properties, see e.g. how to read properties file in scala, and call the file your-application-name.properties. But there is a popular library called simply Config which I'd recommend; as https://github.com/typesafehub/config#standard-behavior says, its standard names for config files are
The convenience method ConfigFactory.load() loads the following (first-listed are higher priority):
system properties
application.conf (all resources on classpath with this name)
application.json (all resources on classpath with this name)
application.properties (all resources on classpath with this name)
reference.conf (all resources on classpath with this name)
The idea is that libraries and frameworks should ship with a reference.conf in their jar. Applications should provide an application.conf , or if they want to create multiple configurations in a single JVM, they could use ConfigFactory.load("myapp") to load their own myapp.conf . (Applications can provide a reference.conf also if they want, but you may not find it necessary to separate it from application.conf .)
As #ashalynd's answer says, in order for any such library to see your configuration files, they should go into src/{main,test}/resources.
I think you are talking about application.conf. It can be found in /src/main/resources or in /src/test/resources (affecting main and test settings, respectively). If application.conf is not present in /src/test/resources, the main application.conf will be used for the tests.
I'm using activator to create my project and I use the command universal:packageBin to generate a .zip file.
I was wondering if it's possible to set up the application to read the configuration file from conf folder instead of the configuration from classpath ( the .conf file inside the jar).
Is it possible? If it's, how can i do it?
See https://www.playframework.com/documentation/2.3.x/ProductionConfiguration for the mechanism of telling a Play application where to pickup its config from.
To tie that in with the universal:packageBin, just pass -Dconfig.file to the script inside the bin directory. e.g. if your application is called foo:
unzip foo-1.0-SNAPSHOT.zip
foo-1.0-SNAPSHOT/bin/foo -Dconfig-file=/path/to/app.conf
I have a custom configuration file for my database, and I want to externalize it when my Play project is packaged into a war.
I tried javaOptions in build.sbt with the -DConfig.file java command, but my file is not detected...
my line : javaOptions in run += "-Dconfig.file=conf/database.conf"
and my deployed war on a Tomcat server is :
/mywar/META-INF/MANIFEST.MF
/mywar/WEB-INF/lib/myplayproject.jar (with other dependencies)
I put my conf file in lib directory : /lib/conf/database.conf
Where am I doing wrong ?
(based on this link : http://www.scala-sbt.org/0.12.3/docs/Detailed-Topics/Forking.html)
If you deploy your war on Tomcat, I don't see how sbt is involved, which may explain why your option is ignored.
Play is not really meant to be deployed as a war. I think the easiest thing for you would be to manually load the config file in your code via the typesafe config library.
When deploying to a container you aren't in control of the JVM process so you can't specify command line options to set system properties.
Your easiest approach here is probably to just rename your config file to reference.conf or application.conf. Those two names are auto-loaded by the configuration library from the classpath.
You could also edit reference.conf or application.conf to add include "database.conf" if you want to keep a separate file.
Otherwise you could manually load a config file, e.g. with ConfigFactory.parseResource to search the classpath.
In addition to including classpath resources you can include files and urls out of the box. Use the include url("http://example.com") or include file("foo.conf") syntax. By hand (in code) ConfigFactory can of course also parse urls and files.
If that isn't flexible enough, it's also possible to customize includes in two ways.
One is to create a custom ConfigIncluder http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigIncluder.html
The other is to create a custom URL protocol in the usual Java way, see Creating custom URI scheme using URI class and http://www.cooljeff.co.uk/2009/12/12/custom-url-protocols-and-multiple-classloaders/ for example. Then use include url("myproto:foobar")
if multiple XML configuration files exist in CLASSPATH, so what will happen? Just pick out a file randomly and ignore others, or the latter configuration overrides the former?
I just tried, does not support multiple configuration files