Static resource reload with akka-http - scala

In short: is it possible to reload static resources using akka-http?
A bit more:
I have Scala project.
I'm using App object to launch my Main
class.
I'm using getFromResourceDirectory to locate my resource
folder.
What I would like to have is to hot-swap my static resources during development.
For example, I have index.html or application.js, which I change and I want to see changes after I refresh my browser without restarting my server. What is the best practise of doing such thing?
I know that Play! allows that, but don't want to base my project on Play! only because of that.

Two options:
Easiest: use the getFromDirectory directive instead when running locally and point it to the path where your files you want to 'hotload' are, it serves them directly from the file system, so every time you change a file and load it through Akka HTTP it will be the latest version.
getFromResourceDirectory loads files from the classpath, the resources are available because SBT copies them into the class directory under target every time you build (copyResources). You could configure sbt using unmanagedClasspath to make it include the static resource directory in the classpath. If you want to package the resources in the artifact when running package however this would require some more sbt-trixery (if you just put src/resources in unmanagedClasspath it will depend on classpath ordering if the copied ones or the modified ones are used).

I couldn't get it to work by adding to unmanagedClasspath so I instead used getFromDirectory. You can use getFromDirectory as a fallback if getFromResourceDirectory fails like this.
val route =
pathSingleSlash {
getFromResource("static/index.html") ~
getFromFile("../website/static/index.html")
} ~
getFromResourceDirectory("static") ~
getFromDirectory("../website/static")
First it tries to look up the file in the static resource directory and if that fails, then checks if ../website/static has the file.

The below code try to find the file in the directory "staticContentDir". If the file is found, it is sent it back to the client. If it is not found, it tries by fetching the file from the directory "site" in the classpath.
The user url is: http://server:port/site/path/to/file.ext
/site/ comes from "staticPath"
val staticContentDir = calculateStaticPath()
val staticPath = "site"
val routes = pathPrefix(staticPath) {
entity(as[HttpRequest]) { requestData =>
val fullPath = requestData.uri.path
encodeResponse {
if (Files.exists(staticContentDir.resolve(fullPath.toString().replaceFirst(s"/$staticPath/", "")))) {
getFromBrowseableDirectory(staticContentDir.toString)
} else {
getFromResourceDirectory("site")
}
}
}
}
I hope it is clear.

Related

How do I overwrite config settings from an included file

I have defined some akka remote settings in my application.conf:
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
warn-about-java-serializer-usage = false
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "myhost"
port = 2561
maximum-frame-size = 256000b
}
}
}
But then I have another program that needs to access other configuration settings from my application.conf. But I need to ignore the akka config settings. So I've tried the following for the second program:
include "application"
akka {}
But the akka settings from application.conf are still being applied. I know this because I get a bind exception on the akka port eventhough there should be no remote akka in my second app
What is the best way for me clear/ignore the akka config settings from my application.conf?
Let's say you want to override the akka.remote.netty.tcp.port in your another.conf, you simply
include "application.conf"
akka.remote.netty.tcp.port = 2562
It will override the netty tcp port while leaving the rest unchanged and inherited
After some experimentation, the best bet IMO is to factor out your application.conf so that your common settings are in their own conf files.
So you might put the common settings in a common-settings.conf and then in your Akka application you would have
include "common-settings"
akka {
// Akka settings here
}
And the other modules that need common-settings can just, in their application.conf:
include "common-settings"
This may work better with a multi-module build. If they're the same, e.g., sbt module, then you'll probably replace the canonical application.conf with akka-application.conf and other-application.conf and point your ActorSystem setup code to akka-application.conf instead of application.conf (which probably shouldn't exist in this scenario, as you'd want bare ConfigFactory.load() calls to fail very quickly (the alternative here is to have different programs fighting over who owns application.conf).
The issue you are facing is actually by design. From the HOCON first page documentation:
Duplicate keys are allowed; later values override earlier, except for object-valued keys where the two objects are merged recursively
Therefore, when you add in your second file an akka {} it is just being merged, and not overwritten.
As I can see it, you have 2 options.
Copying the configuration, and override all properties with the one you actually want. That means, that in the second program, you can add:
akka {
actor {
provider = "new value"
warn-about-java-serializer-usage = false
}
remote {
enabled-transports = ["completely new values"]
netty.tcp {
hostname = "etc..."
port = 2561
maximum-frame-size = 256000b
}
}
}
The other option, which I like less, is to overwrite the object called akka. For doing that, you need to assign to it something that is not an object. Otherwise it will just be merged. For instance, if you add the the second program akka=4, so it will completely remove all of the other values. But! In this case, you have your program to deal with those properties to be missing. That means, that somewhere in your code you will have to write something like (don't forget that config throws on missing):
Try(config.getString("akka.actor.provider")).getOrElse(Do something here)
You have to do that because now akka is a string, and you cannot look into that as an object.

Does SBT to support preemptive auth for downloading packages?

I am running SBT 1.2.8 and my project needs to download packages from a repo on a privately hosted Artifactory instance. My repo is protected by basic auth. After reading a multitude of examples and instructions, I created a credentials.properties file in my repo.
realm=Artifactory Realm
host=artifactory.mycompany.com
username=my_username
password=my_password
I then added the following to my build.sbt file
credentials += Credentials(new File("credentials.properties"))
Then I added the repository to my list of resolvers in resolvers.sbt
"My Company Artifactory" at "https://artifactory.mycompany.com/artifactory/my_private_repo/",
I built my application and was able to download the protected packages just fine.
However, a system administrator at my company requested I turn on the “Hide Existence of Unauthorized Resources” setting in Artifactory. This setting forces Artifactory to return 404 errors when an unauthenticated user tries to access protected resources. Usually in this case, Artifactory returns 401s with a WWW-Authenticate header.
Suddenly, my application was unable to resolve its dependencies. I turned the Artifactory setting off and then back on again and verified this setting was, in fact, the cause of my problems.
It appears as though SBT will not send credentials unless it is challenged with a 401 and a WWW-Authenticate header (with the proper realm). Looking at the docs and GitHub issues for SBT, Ivy, and Coursier, it seems this “preemptive authentication” is not a supported feature.
I spend many hours trying to resolve this in various ways, but I cannot find a solution. Here is what I have tried:
Adding my Artifactory username and password to the repository url, so it looks like https://my_username:my_password#artifactory.mycompany.com/artifactory/my_private_repo/. This worked in my browser and a REST client, but not with SBT.
Omitting the “realm” from my credentials file
Switching to SBT 1.3.9 and trying everything above with the new version.
Does anyone know how I can get SBT to use preemptive HTTP basic auth? It appears both Maven and Gradle support this (see links below), but I cannot find anything in the SBT docs.
Maven support for preemptive auth: https://jfrog.com/knowledge-base/why-does-my-maven-builds-are-failing-with-a-404-error-when-hide-existence-of-unauthorized-resources-is-enabled/
Gradle support for preemptive auth:
https://github.com/gradle/gradle/pull/386/files
I'm almost thinking of setting up a local proxy to send the proper headers Artifactory, and point SBT to use the local proxy as a resolver. However, that seems needlessly cumbersome for developers to use.
you are correct.
You can setup an AbstractRepository. See https://github.com/SupraFii/sbt-google-artifact-registry/blob/master/src/main/scala/ch/firsts/sbt/gar/ArtifactRegistryRepository.scala#L21 for example:
package ch.firsts.sbt.gar
import java.io.File
import java.util
import com.google.cloud.artifactregistry.wagon.ArtifactRegistryWagon
import org.apache.ivy.core.module.descriptor.Artifact
import org.apache.ivy.plugins.repository.AbstractRepository
import org.apache.maven.wagon.repository.Repository
class ArtifactRegistryRepository(repositoryUrl: String) extends AbstractRepository {
val repo = new Repository("google-artifact-registry", repositoryUrl)
val wagon = new ArtifactRegistryWagon()
override def getResource(source: String): ArtifactRegistryResource = {
val plainSource = stripRepository(source)
wagon.connect(repo)
ArtifactRegistryResource(repositoryUrl, plainSource, wagon.resourceExists(plainSource))
}
override def get(source: String, destination: File): Unit = {
val adjustedSource = if (destination.toString.endsWith("sha1"))
source + ".sha1"
else if (destination.toString.endsWith("md5"))
source + ".md5"
else
source
wagon.connect(repo)
wagon.get(adjustedSource, destination)
}
override def list(parent: String): util.List[String] = sys.error("Listing repository contents is not supported")
override def put(artifact: Artifact, source: File, destination: String, overwrite: Boolean): Unit = {
val plainDestination = stripRepository(destination)
wagon.connect(repo)
wagon.put(source, plainDestination)
}
private def stripRepository(fullName: String): String = fullName.substring(repositoryUrl.length + 1)
}

Setting DNS lookup's TimeToLive in Scala Play

I am trying to set the TimeToLive setting for DNS Lookup in my Scala-Play application. I use Play 2.5.9 and Scala 2.11.8 and follow the AWS guide. I tried the following ways:
in application.conf
// Set DNS lookup time-to-live to one minute
networkaddress.cache.ttl=1
networkaddress.cache.negative.ttl=1
in AppModule or EagerSingleton (the code would be similar)
class AppModule() extends AbstractModule {
Security.setProperty("networkaddress.cache.ttl", "1")
Security.setProperty("networkaddress.cache.negative.ttl", "1")
...
}
passing as environment variable:
sbt -Dsun.net.inetaddr.ttl=1 clean run
I have the following piece of test code in the application:
for (i <- 1 to 25) {
System.out.println(java.net.InetAddress.getByName("google.com").getHostAddress())
Thread.sleep(1000)
}
This always prints the same IP address, e.g. 216.58.212.206. To me it looks like none of the approaches specified above have any effect. However, maybe I am testing something else and not actually the value of TTL. Therefore, I have two questions:
what is the correct way to pass a security variable into a Play application?
how to test it?
To change the settings for DNS cache via java.security.Security you have to provide a custom application loader.
package modules
class ApplicationLoader extends GuiceApplicationLoader {
override protected def builder(context: Context): GuiceApplicationBuilder = {
java.security.Security.setProperty("networkaddress.cache.ttl", "1")
super.builder(context)
}
}
When you build this application loader you can enable it in your application.conf
play.application.loader = "modules.ApplicationLoader"
after that you could use your code above and check if the DNS cache is behaving like you set it up. But keep in mind that your system is accessing a DNS server which is caching itself so you wont see change then.
If you want to be sure that you get different addresses for google.com you should use an authority name server like ns1.google.com
If you want to write a test on that you could maybe write a test which requests the address and then waits for the specified amount of time until it resolves again. But with a DNS system out of your control like google.com this could be a problem, if you hit a DNS server with caching.
If you want to write such a check you could do it with
#RunWith(classOf[JUnitRunner])
class DnsTests extends FlatSpec with Matchers {
"DNS Cache ttl" should "refresh after 1 second"
in new WithApplicationLoader(new modules.ApplicationLoader) {
// put your test code here
}
}
As you can see you can put the custom application loader in the context of the application starting behind your test.

How to get started with ESAPI out of a servlet container

Could anyone give some considerations to get started using the ESAPI on a no-web context?
I came with this little test that validates a string with DefaultValidator.isValidCreditCard, but I got some web-container dependency errors.
The following method is consumed from a Junit Test:
#Override
public ValidationErrorList creditCard(String value) {
this.value = value;
ValidationErrorList errorList = new ValidationErrorList();
try {
isValid = validator.isValidCreditCard(null, value, false, errorList);
}catch(Exception ie){
System.out.println(">>> CCValidator: [ " + value + "] " + ie.getMessage());
messages = (ArrayList) errorList.errors();
}
return messages;
}
This is the error that I get (relevant part) of course I'm not running in a container:
Attempting to load ESAPI.properties via file I/O.
Attempting to load ESAPI.properties as resource file via file I/O.
Found in 'org.owasp.esapi.resources' directory: C:\foundation\validation\providers\esapi\ESAPI.properties
Loaded 'ESAPI.properties' properties file
Attempting to load validation.properties via file I/O.
Attempting to load validation.properties as resource file via file I/O.
Found in 'org.owasp.esapi.resources' directory: C:\foundation\validation\providers\esapi\validation.properties
Loaded 'validation.properties' properties file
SecurityConfiguration for Encoder.AllowMixedEncoding not found in ESAPI.properties. Using default: false
SecurityConfiguration for Encoder.AllowMixedEncoding not found in ESAPI.properties. Using default: false
javax/servlet/ServletRequest
java.lang.NoClassDefFoundError: javax/servlet/ServletRequest
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.owasp.esapi.util.ObjFactory.make(ObjFactory.java:74)
at org.owasp.esapi.ESAPI.httpUtilities(ESAPI.java:121)
at org.owasp.esapi.ESAPI.currentRequest(ESAPI.java:70)
at org.owasp.esapi.reference.Log4JLogger.log(Log4JLogger.java:434)
...
Calls to ESAPI..xxxMethods() also raise dependency errors.
Any advice to get started will be appreciate.
Best,
jose
ESAPI has a servlet filter API that requires javax.servlet.ServletRequest to be on the classpath. ESAPI is owned by OWASP --> "Open Web Application Security Project." Therefore, ESAPI is designed with web applications in mind.
If you're not writing a web application, then its either a console application or a rich client application. If you don't expect to use it to connect to the outside world, then the main secure practices you really need to worry about are ensuring that you always use safely parameterized queries, and that any data passed into your application from a source that IS connected to the outside world is properly escaped. For that, the only thing you need is OWASP's encoder project.

Code coverage on Play! project

I have a Play! project where I would like to add some code coverage information. So far I have tried JaCoCo and scct. The former has the problem that it is based on bytecode, hence it seems to give warning about missing tests for methods that are autogenerated by the Scala compiler, such as copy or canEqual. scct seems a better option, but in any case I get many errors during tests with both.
Let me stick with scct. I essentially get errors for every test that tries to connect to the database. Many of my tests load some fixtures into an H2 database in memory and then make some assertions. My Global.scala contains
override def onStart(app: Application) {
SessionFactory.concreteFactory = Some(() => connection)
def connection() = {
Session.create(DB.getConnection()(app), new MySQLInnoDBAdapter)
}
}
while the tests usually are enclosed in a block like
class MySpec extends Specification {
def app = FakeApplication(additionalConfiguration = inMemoryDatabase())
"The models" should {
"be five" in running(app) {
Fixtures.load()
MyModels.all.size should be_==(5)
}
}
}
The line running(app) allows me to run a test in the context of a working application connected to an in-memory database, at least usually. But when I run code coverage tasks, such as scct coverage:doc, I get a lot of errors related to connecting to the database.
What is even more weird is that there are at least 4 different errors, like:
ObjectExistsException: Cache play already exists
SQLException: Attempting to obtain a connection from a pool that has already been shutdown
Configuration error [Cannot connect to database [default]]
No suitable driver found for jdbc:h2:mem:play-test--410454547
Why is that launching tests in the default configuration is able to connect to the database, while running in the context of scct (or JaCoCo) fails to initialize the cache and the db?
specs2 tests run in parallel by default. Play disables parallel execution for the standard unit test configuration, but scct uses a different configuration so it doesn't know not to run in parallel.
Try adding this to your Build.scala:
.settings(parallelExecution in ScctPlugin.ScctTest := false)
Alternatively, you can add sequential to the beginning of your test classes to force all possible run configurations to run sequentially. I've got both in my files still, as I think I had some problems with the Build.scala solution at one point when I was using an early release candidate of Play.
A better option for Scala code coverage is Scoverage which gives statement line coverage.
https://github.com/scoverage/scalac-scoverage-plugin
Add to project/plugins.sbt:
addSbtPlugin("com.sksamuel.scoverage" % "sbt-scoverage" % "1.0.1")
Then run SBT with
sbt clean coverage test
You need to add sequential in the beginning of your Specification.
class MySpec extends Specification {
sequential
"MyApp" should {
//...//
}
}