I'm using play and I want to specifiy the port in application.conf because I'm going to deploy it to cloud.
I've been reading post regarding this question but all answers are about putting it when typing play start. Is there really a way to add it in application.conf?
thanks!
Seems there is no way to do that.
That is how play's Netty Server initialised:
val server = new NettyServer(
new StaticApplication(applicationPath),
Option(System.getProperty("http.port")).map(Integer.parseInt(_)).getOrElse(9000),
Option(System.getProperty("https.port")).map(Integer.parseInt(_)),
Option(System.getProperty("http.address")).getOrElse("0.0.0.0")
)
So it using system property or "9000" if the first one was not set.
In our production env we just pass it within args "-Dhttp.port=<port>".
What are the complexities with your cloud?
Maybe I am missing something but have you tried to specify http.port=80 in your application.conf?
Related
When I try to read a parquet file from a specified location like /test with spark.read.parquet() i get an error saying file://test does not exist. When I add the core-site.xml as a resource in code with
sc.hadoopConfiguration.addResource(new Path(<path-to-core-site.xml>))
it does look in the hdfs. However I don't want to add the resource in code. My question is how do I make sure spark reads the core-site.xml and uses hdfs as default file system.
I've setup an ubuntu 18.04.2LTS server with hadoop 3, spark 2.4.2 and yarn as resourcemanager in a virtual machine. I've configured the core-site.xml with fs.defaultFS set to hdfs://localhost:9000.
I've also configured the HADOOP_CONF_DIR in the bash file.
Couple of options
1. Make sure that core-site.xml is available in driver's classpath. This way the core-site.xml will get loaded automatically.
2. If only setting the default filesystem uri is the requirement, we can set this in spark-defaults.conf or in the SparkConf object created for the application using spark.hadoop.fs.defaultFS and set its value to hdfs:///
Well its a generic question with many possible answers.
Ill try to answer as best as I can :
https://spark.apache.org/docs/latest/configuration.html#inheriting-hadoop-cluster-configuration
It is explained in the link you can set the enviroment variable HADOOP_CONF_DIR to the dir that contains the core-site.xml and as long as you dont override it in spark-env.sh or something it should work.
BTW are you sure you did an export on the HADOOP_CONF_DIR because I know from experience it works with Spark 2.4.2 so if you think the core-site isnt loaded its probably because you didnt define the variable correctly or maybe your spark-env.sh masks your previous definition.
I use the following adjustments for local development in the application.properties:
myproject.mongodb.databaseName=databaseName
myproject.mongodb.databaseAddress=serverName.mycompany.at
myproject.mongodb.databasePort=27017
myproject.mongodb.username=myProjectName
myproject.mongodb.password=asdfasdf
myproject.mongodb.mappingBasePackage=domain
Now I will use the app in a productiv way and therefore need SSL like I can adjust with RoboMongo:
Does anyone know what I have to set in my application.properties file?
As far as I know the only way to connect with options (without extra code) is to use the connection string/uri. It would look something like this:
spring.data.mongodb.uri=mongodb://myProjectName:asdfasdf#serverName.mycompany.at:27017/databaseName?ssl=true
I could set http.port in applicaton.conf by using playframework1.2.7
like this
http.port = 9020
jpda.port = 8020
also jdpa.port.
But in play2.4.
I cannot set http.port in application.conf like this.
I know that I can do like this when I run this project.
activator "run 9020"
But it is too troublesome for me.
If you have some ideas,
please share your idea.
You cannot specify port in aaplication.conf during run mode (but this can be used while deploying).
In run mode the HTTP server part of Play starts before the application has been compiled. This means that the HTTP server cannot access the application.conf file when it starts. If you want to override HTTP server settings while using the run command you cannot use the application.conf file. Instead, you need to either use system properties or the devSettings setting shown above.
Source: https://www.playframework.com/documentation/2.4.x/Configuration#HTTP-server-settings-in-application.conf
Also look at full server configuration options
https://www.playframework.com/documentation/2.4.x/ProductionConfiguration#Server-configuration-options
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my requirement.
I have a REST end point where I expose API to trigger Jobs, I have used Spring4.0 for Rest Implementation. Now going ahead I thought of implementing Jobs as Service in Spring where I would submit Job programmatically, meaning when the endpoint is triggered, with given parameters I would trigger the job.
I have now few design options.
Similar to the below written job, I need to maintain several Jobs called by a Abstract Class may be JobScheduler .
/*Can this Code be abstracted from the application and written as
as a seperate job. Because my understanding is that the
Application code itself has to have the addJars embedded
which internally sparkContext takes care.*/
SparkConf sparkConf = new SparkConf().setAppName("MyApp").setJars(
new String[] { "/path/to/jar/submit/cluster" })
.setMaster("/url/of/master/node");
sparkConf.setSparkHome("/path/to/spark/");
sparkConf.set("spark.scheduler.mode", "FAIR");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.setLocalProperty("spark.scheduler.pool", "test");
// Application with Algorithm , transformations
extending above point have multiple versions of jobs handled by service.
Or else use an Spark Job Server to do this.
Firstly, I would like to know what is the best solution in this case, execution wise and also scaling wise.
Note : I am using a standalone cluster from spark.
kindly help.
It turns out Spark has a hidden REST API to submit a job, check status and kill.
Check out full example here: http://arturmkrtchyan.com/apache-spark-hidden-rest-api
Just use the Spark JobServer
https://github.com/spark-jobserver/spark-jobserver
There are a lot of things to consider with making a service, and the Spark JobServer has most of them covered already. If you find things that aren't good enough, it should be easy to make a request and add code to their system rather than reinventing it from scratch
Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.
Here is a good client that you might find helpful: https://github.com/ywilkof/spark-jobs-rest-client
Edit: this answer was given in 2015. There are options like Livy available now.
Even I had this requirement I could do it using Livy Server, as one of the contributor Josemy mentioned. Following are the steps I took, hope it helps somebody:
Download livy zip from https://livy.apache.org/download/
Follow instructions: https://livy.apache.org/get-started/
Upload the zip to a client.
Unzip the file
Check for the following two parameters if doesn't exists, create with right path
export SPARK_HOME=/opt/spark
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
Enable 8998 port on the client
Update $LIVY_HOME/conf/livy.conf with master details any other stuff needed
Note: Template are there in $LIVY_HOME/conf
Eg. livy.file.local-dir-whitelist = /home/folder-where-the-jar-will-be-kept/
Run the server
$LIVY_HOME/bin/livy-server start
Stop the server
$LIVY_HOME/bin/livy-server stop
UI: <client-ip>:8998/ui/
Submitting job:POST : http://<your client ip goes here>:8998/batches
{
"className" : "<ur class name will come here with package name>",
"file" : "your jar location",
"args" : ["arg1", "arg2", "arg3" ]
}
I'm working on a CherryPy application based on what I found on that BitBucket repository.
As in this example, there is two config files, server.cfg (aka "global") and app.cfg.
Both config files are loaded in the serve.py file :
# Update the global settings for the HTTP server and engine
cherrypy.config.update(os.path.join(self.conf_path, "server.cfg"))
# ...
# Our application
from webapp.app import Twiseless
webapp = Twiseless()
# Let's mount the application so that CherryPy can serve it
app = cherrypy.tree.mount(webapp, '/', os.path.join(self.conf_path, "app.cfg"))
Now, I'd like to add the Database configuration.
My first thought was to add it in the server.cfg (is this the best place? or should it be located in app.cfg ?).
But if I add the Database configuration in the server.cfg, I don't know how to access it.
Using :
cherrypy.request.app.config['Database']
Works only if the [Database] parameter is in the app.cfg.
I tried to print cherrypy.request.app.config, and it shows me only the values defined in app.cfg, nothing in server.cfg.
So I have two related question :
Is it best to put the database connection in the server.cfg or app.cfg file
How to access server.cfg configuration (aka global) in my code
Thanks for your help! :)
Put it in the app config. A good question to help you decide where to put such things is, "if I mounted an unrelated blog app at /blogs on the same server, would I want it to share that config?" If so, put it in server config. If not, put it in app config.
Note also that the global config isn't sectioned, so you can't stick a [Database] section in there anyway. Only the app config allows sections. If you wanted to stick database settings in the global config anyway, you'd have to consider config entry names like "database_port" instead. You would then access it directly by that name: cherrypy.config.get("database_port").