Adding a Configuration at runtime - scala

I'd like to add a configuration within a plugin at runtime and probably also register it at projectConfigurations. Is there a possibility to manipulate the state in such a way? I assume this needs to be done in a command, and the solution is buried somewhere deep inside of the AttributeMap, but I'm not sure how to approach that beast.
def addConfiguration = Command.single( "add-configuration" )( ( state, configuration ) => {
val extracted = Project extract state
val c = configurations.apply( configuration )
// Add c to the state ...
???
} )
Background
I'm working on a sbt-slick-codegen plugin, which has multi-database support as an essential feature. Now to configure databases in sbt, I took a very simple approach: everything is stuffed into a Map.
databases in SlickCodegen ++= Map(
"default" -> Database(
Authentication(
url = "jdbc:postgresql://localhost:5432/db",
username = Some( "db_user_123" ),
password = Some( "p#assw0rd" )
),
Driver(
jdbc = "org.postgresql.Driver",
slick = slick.driver.PostgresDriver,
user = Some( "com.example.MySlickProfile" )
),
container = "Tables",
generator = new SourceCodeGenerator( _ ),
identifier = Some( "com.example" ),
excludes = Seq( "play_evolutions" ),
cache = true
),
"user_db" -> Database( ... )
)
Now, this works, but defeats the purpose of sbt, is hard to setup, difficult to maintain and update. I'd instead prefer configuration to work like this:
container in Database := "MyDefaultName",
url in Database( "backup" ) := "jdbc:postgresql://localhost:5432/database",
cache in Database( "default" ) := false
And also this approach is working on the current development branch, but only for Database( x )-configurations that are known at compile time, which leads to the next problem:
On top of that, I'm planning to add a module with PlayFramework support. The idea is to have a setting SettingKey[Seq[Config]]( "configurations" ) that accepts Typesafe Config instances, reads the slick configurations from them and generates appropriate Database( x )-configurations. But for this step I am out of ideas.

Related

Programmatic configuration for Great Expectations

I'm looking into integrating a validation framework to an existing PySpark project. There are a lot of examples how to configure Great Expectations using JSON/YAML files in official documentation. However, in my case table schemas are defined as Python classes and I'm aiming to keep the validation definitions in these classes.
When playing around, I noticed this kind of pattern can be used to validate single expectations without any config files:
spark = SparkSession.builder.master("local[*]").getOrCreate()
df = spark.createDataFrame([
Row(x=1, y="foo"),
Row(x=2, y=None),
])
ds = SparkDFDataset(df)
expectation: ExpectationValidationResult = ds.expect_column_values_to_not_be_null("y")
print(expectation.success)
where expectation.success is either False or True. However, I'm aiming to build expectation suites and generating reports using programmatic configuration but can't find any references how to do it. This is what I tried to hack but it leads to a runtime exception:
ds.append_expectation(ExpectationConfiguration(
expectation_type="expect_column_values_to_not_be_null",
kwargs={'column': 'y', 'result_format': 'BASIC'},
))
engine = SparkDFExecutionEngine(
force_reuse_spark_context=True,
)
validator = Validator(
execution_engine=engine,
expectation_suite=ds.get_expectation_suite(),
)
res = validator.validate()
Any pointers on how to configure Great Expectations without config files (or minimal files) are highly appreciated!
Batches are required for validating expectation suites. Here is a working example:
spark = SparkSession.builder.master("local[*]").getOrCreate()
df = spark.createDataFrame([
Row(x=1, y="foo"),
Row(x=2, y=None),
])
engine = SparkDFExecutionEngine(
force_reuse_spark_context=True,
)
validator = Validator(
execution_engine=engine,
expectation_suite=ExpectationSuite(
expectation_suite_name="my_suite",
expectations=[
ExpectationConfiguration(
expectation_type="expect_column_values_to_not_be_null",
kwargs={"column": "y", "result_format": "BASIC"},
),
ExpectationConfiguration(
expectation_type="expect_column_values_to_not_be_null",
kwargs={"column": "x", "result_format": "BASIC"},
)
]
),
batches=[
Batch(
data=df,
batch_definition=BatchDefinition(
datasource_name="foo",
data_connector_name="foo",
data_asset_name="foo",
batch_identifiers=IDDict(ge_batch_id=str(uuid.uuid1())),
),
),
],
)
res = validator.validate()

File upload, declaration via apiOperation (Swagger and Scalatra 2.6)

There is an existing project that uses Scalatra (2.6) and Swagger:
scalaMajorVersion = '2.12'
scalaVersion = "${scalaMajorVersion}.8"
scalatraVersion = "${scalaMajorVersion}:2.6.4"
compile "org.scalatra:scalatra-swagger_${scalatraVersion}"
I easily could add a new end point like:
get ("/upload", op[String]( // op finally invokes apiOperation
name = "Test method",
params = List(
query[Long]("id" -> "ID"),
query[String]("loginName" -> "login name")
),
authorizations = List(Permission.xxxxx.name)
)) {
...
}
but I cannot upload a file.
I expect to see a file selector button, but instead I see a single-line edit field.
(There are numerous things I'm uncertain about: form or file, [String] or [FileItem], which trait(s), what kind of initialization, etc.)
In the existing code I found a comment that someone could not get swagger to handle file upload. At the same time, I read that Scalatra and Swagger can do that, not all versions of them, but it looks like the version used in the project should be able to do that.
I could find code examples with yml/json interface definitions, but in the project there is no yml, only the apiOperation-based stuff.
Is there a working example using Scalatra 2.6, Swagger, and apiOperation?
I managed to get the file chooser (file selector, "Browse") button; there was no predefined constant (like DataType.String) for that. After I used DataType("File"), everything else just worked.
https://docs.swagger.io/spec.html says:
4.3.5 File
The File (case sensitive) is a special type used to denote file
upload. Note that declaring a model with the name File may lead to
various conflicts with third party tools and SHOULD be avoided.
When using File, the consumes field MUST be "multipart/form-data", and
the paramType MUST be "form".
post ("/uploadfile", op[String]( // op finally invokes apiOperation
name = "Upload File",
params = List(
new Parameter(
`name` = "kindaName",
`description` = Some("kindaDescription2"),
`type` = DataType("File"), // <===== !!!!!!
`notes` = Some("kindaNotes"),
`paramType` = ParamType.Form, // <===== !!
`defaultValue` = None,
`allowableValues` = AllowableValues.AnyValue,
`required` = true
)
),
consumes = List("multipart/form-data"), // <===== !!
...
)) {
val file: FileItem = fileParams("kindaName") // exception if missing
println("===========")
println("file: " + file)
println("name: " + file.getName + " size:"+file.getSize+" fieldName:"+file.getFieldName+ " ContentType:"+file.getContentType+" Charset:"+file.getCharset)
println("====vvv====")
io.copy(file.getInputStream, System.out)
println("====^^^====")
val file2: Option[FileItem] = fileParams.get("file123") // None if missing, and it is missing
println("file2: " + file2)
PS the apiOperation stuff is called "annotations".

How can I print all the settings in Test configuration for a project build using SBT?

I have a scala-js project, adding a particular library dependency, is affecting the way project test cases are running. Without the library dependency everything's fine, the moment I add them, tests doesn't execute. I want to check all sbt settings, if those are getting affected. Is there any way I can print all settings and check?
BuildStructure.data seems to give access to all the settings by scope. We could access it by defining a custom command printAllTestSettings like so:
def printAllTestSettings = Command.command("printAllTestSettings") { state =>
val structure = Project.extract(state).structure
val testScope =
Scope(
Select(ProjectRef(new File("/home/mario/sandbox/hello-world-scala/"), "root")),
Select(ConfigKey("test")),
Zero,
Zero
)
structure
.data
.keys(testScope)
.foreach(key => println(s"${key.label} = ${structure.data.get(testScope, key).get}"))
state
}
commands ++= Seq(printAllTestSettings)
Here is output snippet:
...
managedSourceDirectories = List(/home/mario/sandbox/hello-world-scala/target/scala-2.12/src_managed/test)
managedResourceDirectories = List(/home/mario/sandbox/hello-world-scala/target/scala-2.12/resource_managed/test)
testLoader = Task((taskDefinitionKey: ScopedKey(Scope(Select(ProjectRef(file:/home/mario/sandbox/hello-world-scala/,root)), Select(ConfigKey(test)), Zero, Zero),testLoader)))
packageBin = Task((taskDefinitionKey: ScopedKey(Scope(Select(ProjectRef(file:/home/mario/sandbox/hello-world-scala/,root)), Select(ConfigKey(test)), Zero, Zero),packageBin)))
...

Remove or Exclude WatchSource in sbt 1.0.x

Overview
After looking around the internet for a while, I have not found a good way to omit certain folders from being watched by sbt 1.0.x in a Play Framework application.
Solutions posted for older versions of sbt:
How to exclude a folder from compilation
How to not watch a file for changes in Play Framework
There are a few more, but all more or less the same.
And the release notes for 1.0.2 show that the += and ++= behavior was maintained, but everything else was dropped.
https://www.scala-sbt.org/1.x/docs/sbt-1.0-Release-Notes.html
Source code verifies: https://www.scala-sbt.org/1.0.4/api/sbt/Watched$.html
Would love to see if anyone using sbt 1.0.x has found a solution or workaround to this issue. Thanks!
Taking the approach of how SBT excludes managedSources from watchSources I was able to omit a custom folder from being watched like so:
watchSources := {
val directoryToExclude = "/Users/mgalic/sandbox/scala/scala-seed-project/src/main/scala/dirToExclude"
val filesToExclude = (new File(directoryToExclude) ** "*.scala").get.toSet
val customSourcesFilter = new FileFilter {
override def accept(pathname: File): Boolean = filesToExclude.contains(pathname)
override def toString = s"CustomSourcesFilter($filesToExclude)"
}
watchSources.value.map { source =>
new Source(
source.base,
source.includeFilter,
source.excludeFilter || customSourcesFilter,
source.recursive
)
}
},
Here we use PathFinder to get all the *.scala sources from directoryToExclude:
val filesToExclude = (new File(directoryToExclude) ** "*.scala").get.toSet
Then we create customSourcesFilter using filesToExclude, which we then add to every current WatchSource:
watchSources.value.map { source =>
new Source(
...
source.excludeFilter || customSourcesFilter,
...
)
}
Note the above solution is just something that worked for me, that is, I do not know what is the recommend approach of solving this problem.

How to fork the jvm for each test in sbt

I am working with some classes that (for some reason) can only be used once within a single VM. My test cases work if I run them individually (fork := true) enabled in my sbt settings.
If I run more than one of these tests, they fail with an exception that has to with a thread executor rejecting a task (it's most likely closed). It would be very time consuming to find out what causes the problem and even if I find the problem, I might not be able to resolve it (I do not have access to the source code).
I am currently using the specs2 test framework, but any test framework using sbt would be acceptable.
Is there any test framework for sbt that is capable of running each test in a jvm fork?
Thoughts or ideas on possible other solutions are of course welcome.
It turns out this is fairly easy to achieve. The documentation is sufficient and can be found at Testing - Forking tests
// Define a method to group tests, in my case a single test per group
def singleTests(tests: Seq[TestDefinition]) =
tests map { test =>
new Group(
name = test.name,
tests = Seq(test),
runPolicy = SubProcess(javaOptions = Seq.empty[String]))
}
// Add the following to the `Project` settings
testGrouping in Test <<= definedTests in Test map singleTests
Using non-deprecated syntax:
testGrouping in Test := (definedTests in Test).value map { test =>
Tests.Group(name = test.name, tests = Seq(test), runPolicy = Tests.SubProcess(
ForkOptions(
javaHome.value,
outputStrategy.value,
Nil,
Some(baseDirectory.value),
javaOptions.value,
connectInput.value,
envVars.value
)))
}
2023 proper syntax:
Test / testGrouping := (Test / definedTests).value map { test =>
Tests.Group(name = test.name, tests = Seq(test), runPolicy = Tests.SubProcess(
ForkOptions(
javaHome = javaHome.value,
outputStrategy = outputStrategy.value,
bootJars = Vector.empty,
workingDirectory = Some(baseDirectory.value),
runJVMOptions = javaOptions.value.toVector,
connectInput = connectInput.value,
envVars = envVars.value
)))