I need find filename in folder (in build.sbt), and add in artifact list.
//in build.sbt
val myZipTask = taskKey[File]("return the bundle:dist-zip file")
myZipTask := {
val filesArray = new java.io.File("/target/bundle").listFiles()
//here need to find the file name by coincidence and convert to a string
file(fileName)
}; addArtifact( Artifact("bundle", "zip", "zip"), myZipTask)
I tried this option
//in build.sbt
val myZipTask = taskKey[File]("return the bundle:dist-zip file")
myZipTask := {
import java.io.File
def getListOfFiles(dir: String): List[String] = {
val file = new File(dir)
file.listFiles.filter(_.isFile)
.filter(_.getName.startsWith("startName"))
.map(_.getPath).toList
}
getListOfFiles("/target/bundle")
}; addArtifact( Artifact("bundle", "zip", "zip"), myZipTask)
And sbt return me error:
build.sbt: error: type mismatch;
found : List[String]
required: sbt.File
(which expands to) java.io.File
getListOfFiles("/target/bundle")
^
Check the documentation on Path Finders:
val finder: PathFinder = target.value / "bundle"
You can add * "startName*" if you want to filter by prefix. If you call finder.get, it will return you a Seq[File], so this is what you wanted from your getListOfFiles.
But the problem with your code is that you need to return one file, not a list. You could either output an error if the file doesn't exist:
finder.get.headOption.getOrElse {
sys.error("Couldn't find bundle dist-zip file")
}
or change you task type to Option[File], return finder.get.headOption and add the artifact only if the file is there:
myZipTask.value.foreach { zipFile =>
addArtifact(Artifact("bundle", "zip", "zip"), zipFile)
}
This foreach could work even for multiple files if that's an option in your usecase.
Related
My project stopped working from one day to another, without any changes in the project. I'm suspecting one of the dependencies updated, but it's unclear from the error message.
[warn] Merging 'META-INF/aop.xml' with strategy 'aopMerge'
[error] org.xml.sax.SAXParseExceptionpublicId: -//AspectJ//DTD//EN; systemId: http://www.eclipse.org/aspectj/dtd/aspectj.dtd; lineNumber: 1; columnNumber: 2; The markup declarations contained or pointed to by the document type declaration must be well-formed.
The aopMerge strategy in my build.sbt is
val aopMerge: MergeStrategy = new MergeStrategy {
val name = "aopMerge"
import scala.xml._
import scala.xml.dtd._
def apply(tempDir: File, path: String, files: Seq[File]): Either[String, Seq[(File, String)]] = {
val dt = DocType("aspectj", PublicID("-//AspectJ//DTD//EN", "https://www.eclipse.org/aspectj/dtd/aspectj.dtd"), Nil)
val file = MergeStrategy.createMergeTarget(tempDir, path)
val xmls: Seq[Elem] = files.map(XML.loadFile)
val aspectsChildren: Seq[Node] = xmls.flatMap(_ \\ "aspectj" \ "aspects" \ "_")
val weaverChildren: Seq[Node] = xmls.flatMap(_ \\ "aspectj" \ "weaver" \ "_")
val options: String = xmls.map(x => (x \\ "aspectj" \ "weaver" \ "#options").text).mkString(" ").trim
val weaverAttr = if (options.isEmpty) Null else new UnprefixedAttribute("options", options, Null)
val aspects = new Elem(null, "aspects", Null, TopScope, false, aspectsChildren: _*)
val weaver = new Elem(null, "weaver", weaverAttr, TopScope, false, weaverChildren: _*)
val aspectj = new Elem(null, "aspectj", Null, TopScope, false, aspects, weaver)
XML.save(file.toString, aspectj, "UTF-8", xmlDecl = false, dt)
IO.append(file, IO.Newline.getBytes(IO.defaultCharset))
Right(Seq(file -> path))
}
}
I tried changing http://www.eclipse.org/aspectj/dtd/aspectj.dtd into https://www.eclipse.org/aspectj/dtd/aspectj.dtd based on a similar question on StackOverflow. Unfortunately, I'm getting the same error, even with http:// instead of https://. Note that I did run sbt reload after the change.
How can I find out why this is happening? And what can I do to solve this?
The workaround I used was generating a new file with the HTTPS url based on the files passed in files: Seq[File]) parameter. Then changing val xmls: Seq[Elem] = files.map(XML.loadFile) to use this new files.
val xmls: Seq[Elem] = files.map(generateFileWithSecureDtdUrl).map(XML.loadFile)
with the following declared on build.sbt
def generateFileWithSecureDtdUrl(originalFile: File): File = {
val fixedFileName = originalFile.getPath + ".fixed"
val ps: PrintStream = new PrintStream(fixedFileName) {
val originalFileSource: BufferedSource = Source.fromFile(originalFile)
originalFileSource
.getLines()
.map { line =>
if (line.contains("DOCTYPE") && line.contains("http://www.eclipse.org/aspectj/dtd/aspectj.dtd"))
line.replace("http://www.eclipse.org/aspectj/dtd/aspectj.dtd", "https://www.eclipse.org/aspectj/dtd/aspectj.dtd")
else
line
}
.foreach(line => write(line.getBytes("UTF8")))
originalFileSource.close()
}
ps.close()
new File(fixedFileName) }
I have a directory structure like this:
a
/one
hey.class
hey.tasty
you.class
you.tasty
/two
foo.class
foo.tasty
/three
bar.class
bar.tasty
b
/one
/two
/three
I need a way to copy all the .tasty files from their respective places in /a to their corresponding places in /b in a sbt task.
You can create the sbt task and pass the original and destination folders in the following manner:
val copyTasties = inputKey[Unit]("Copy .tasty files")
copyTasties := {
val userInput = Def.spaceDelimited().parsed
if (userInput.size != 2) {
throw new IllegalArgumentException("Original and target directories should be define!")
}
val from = Paths.get(userInput.head)
val to = Paths.get(userInput.last)
Files
.walk(from)
.filter(Files.isRegularFile(_))
.filter(path => path.toString.endsWith("tasty"))
.forEach { original =>
val relative = from.relativize(original)
val destination = to.resolve(relative)
IO.copyFile(original.toFile, destination.toFile)
}
}
Then you can invoke it like this:
copyTasties C:\\Dev\\sandbox\\a C:\\Dev\\sandbox\\b
If the original and destination are stable(for example they are directories inside the project) you can rewrite the task:
import java.nio.file.{Files, Paths}
import sbt._
val copyTastiesHardcoded = taskKey[Unit]("Copy .tasty files")
copyTastiesHardcoded := {
val baseDir = baseDirectory.value.toPath
val from = baseDir.resolve("a")
val to = baseDir.resolve("b")
Files
.walk(from)
.filter(Files.isRegularFile(_))
.filter(path => path.toString.endsWith("tasty"))
.forEach { original =>
val relative = from.relativize(original)
val destination = to.resolve(relative)
IO.copyFile(original.toFile, destination.toFile)
}
}
and invoke it without arguments
copyTastiesHardcoded
There are some useful utility methods in sbt that will help you find files and copy them around. Unfortunately the use a wild mixture of the traditional java.io.File and the more modern java.nio.file.Path, so you need to do conversions between them:
val copyFiles = taskKey[Unit]("copy files")
copyFiles := {
val inputDir = baseDirectory.value.toPath / "a"
val files: Seq[(Path, FileAttributes)] =
FileTreeView.default.list(inputDir.toGlob / RecursiveGlob / "*.tasty")
val outputDir = baseDirectory.value.toPath / "b"
IO.copy {
files.map { case (p, _) =>
p.toFile -> outputDir.resolve(inputDir.relativize(p)).toFile
}
}
}
Note that I'm using a glob here to find the files:
https://www.scala-sbt.org/1.x/docs/Globs.html
I have a spark/scala project named as Omega
I have a conf file inside Omega/conf/omega.config
I use API's from typesafe to load the config file from conf/omega.config.
It was working fine and I was able to read the respective value for each key
Now today, For the first time I added some more key-value pairs in my omega.config file and tried to retrieve them from my scala code. It throws
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'job_name'
This issue started happening after adding new value for the key job_name in my omega.config file.
Also I am not able to read the newly added -key-values, I am still able to read all old values using config. getString method
I am building my spark/scala application using maven.
Omega.config
input_path="/user/cloudera/data
user_name="surender"
job_name="SAMPLE"
I am Not able to access the recently added key "job_name" alone
package com.pack1
import com.pack2.ApplicationUtil
object OmegaMain {
val config_loc = "conf/omega.config"
def main(args: Array[String]): Unit = {
val config = ApplicationUtil.loadConfig(config_loc)
val jobName = ApplicationUtil.getFromConfig(config,"job_name")
}
}
package com.pack2
import com.typesafe.config.{Config, ConfigFactory}
object ApplicationUtil {
def loadConfig(filePath:String):Config={
val config = ConfigFactory.parseFile(new File(filePath))
config
}
def getFromConfig(config:Config,jobName:String):String={
config.getString(jobName)
}
}
Could some one help me what went wrong?
You can try something like:
def loadConfig(filename: String, syntax: ConfigSyntax): Config = {
val in: InputStream = getClass.getResourceAsStream(filename)
if (in == null) return null
val file: File = File.createTempFile(String.valueOf(in.hashCode()), ".conf")
file.deleteOnExit()
val out: FileOutputStream = new FileOutputStream(file)
val buffer: Array[Byte] = new Array(1024)
var bytesRead: Int = in.read(buffer)
while (bytesRead != -1) { out.write(buffer, 0, bytesRead); bytesRead = in.read(buffer) }
out.close()
val conf: Config = ConfigFactory.parseFile(file, ConfigParseOptions.defaults().setSyntax(syntax).setAllowMissing(false).setOriginDescription("Merged with " + filename))
conf
}
filename is some file path in the classpath. If you want to update this method to taking some external file into account, change update the 4th with val file: File = new File("absolute Path of he file")
I am guessing the file isn't on the classpath after you build with Maven.
Since you are using Maven to build a jar, you need your omega.config to be in the classpath. This means that you either have to put it into src/main/resources by default or explicitly tell Maven to add conf to the default resources classpath.
I am trying to write an SBT task that may be used like this:
> deploy key1=value1 key2=value2 ...
and does the following:
reads a default properties file
adds the parsed keys and values to the properties object
writes the new properties object to resources/config.properties
packages a fat .jar withsbt-assembly
writes the default properties to resources/config.properties
I have been trying to achieve it with Def.sequential, but I cant seem to find a way to use it withinputKey. Mybuild.sbtlooks like this:
val config = inputKey[Unit] (
"Set configuration options before deployment.")
val deploy = inputKey[Unit](
"assemble fat .jar with configuration options")
val defaultProperties = settingKey[Properties](
"default application properties.")
val propertiesPath = settingKey[File]("path to config.properties")
val writeDefaultProperties = taskKey[Unit]("write default properties file.")
val parser = (((' ' ~> StringBasic) <~ '=') ~ StringBasic).+
lazy val root = (project in file("."))
.settings(
propertiesPath := {
val base = (resourceDirectory in Compile).value
base / "config.properties"
},
defaultProperties := {
val path = propertiesPath.value
val defaultConfig = new Properties
IO.load(defaultConfig, path)
defaultConfig
},
config := {
val path = propertiesPath.value
val defaultConfig = defaultProperties.value
val options = parser.parsed
val deployConfig = new Properties
deployConfig.putAll(defaultConfig)
options.foreach(option =>
deployConfig
.setProperty(option._1, option._2))
IO.write(deployConfig, "", path)
},
writeDefaultProperties := {
val default = defaultProperties.value
val path = propertiesPath.value
IO.write(default, "", path)
},
deploy := Def.sequential(
config.parsed, // does not compile
assembly,
writeDefaultProperties),
...)
Can I makeDef.sequential work with input keys, or do I need to do something more involved?
See Defining a sequential task with Def.sequential. It uses scalastyle as an example:
(scalastyle in Compile).toTask("")
lazy val buildDb = taskKey[Unit]("Initializes the database")
buildDb := {
(compile in Compile).value
val s: TaskStreams = streams.value
s.log.info("Building database")
try {
...
} catch {
case e: Throwable =>
sys.error("Failed to initialize the database: " + e.getMessage)
}
s.log.info("Finished building database")
}
This produces the following error
C:\work\server\build.sbt:98: error: type mismatch;
found : Unit
required: T
s.log.info("Finished building database")
^
[error] Type error in expression
But if I define it like this lazy val buildDb = taskKey[String]("Initializes the database") and then add to the last line in the task "Happy end!" string everything seem to work. Am I to blame, or something wrong with the macro?
The same happened to me. I was able to fix the issue e.g. by adding a : TaskKey[Unit] to the taskKey definition. Here are my findings for sbt 0.13.5:
The following definition is OK (it seems that it is pure luck that this is OK):
lazy val collectJars = taskKey[Unit]("collects JARs")
collectJars := {
println("these are my JARs:")
(externalDependencyClasspath in Runtime).value foreach println
}
The following definition (the same as above without the first println) yields the same error "found: Unit, required: T":
lazy val collectJars = taskKey[Unit]("collects JARs")
collectJars := {
(externalDependencyClasspath in Runtime).value foreach println
}
My findings are that this is definitely something magical: For example, if I indent the line lazy val collectJars = ... by one blank, then it compiles. I would expect (but have not checked) that .sbt and .scala build definitions also behave differently.
However, if you add the type signature, it seems to always compile:
lazy val collectJars: TaskKey[Unit] = taskKey[Unit]("collects JARs")
collectJars := {
(externalDependencyClasspath in Runtime).value foreach println
}
Last but not least: The issue seems to be specific for TaskKey[Unit]. Unit tasks are not a good idea - in your example, you could at least return Boolean (true for success / false for failure).