Can I relative-copy files in sbt? - scala

I have a directory structure like this:
a
/one
hey.class
hey.tasty
you.class
you.tasty
/two
foo.class
foo.tasty
/three
bar.class
bar.tasty
b
/one
/two
/three
I need a way to copy all the .tasty files from their respective places in /a to their corresponding places in /b in a sbt task.

You can create the sbt task and pass the original and destination folders in the following manner:
val copyTasties = inputKey[Unit]("Copy .tasty files")
copyTasties := {
val userInput = Def.spaceDelimited().parsed
if (userInput.size != 2) {
throw new IllegalArgumentException("Original and target directories should be define!")
}
val from = Paths.get(userInput.head)
val to = Paths.get(userInput.last)
Files
.walk(from)
.filter(Files.isRegularFile(_))
.filter(path => path.toString.endsWith("tasty"))
.forEach { original =>
val relative = from.relativize(original)
val destination = to.resolve(relative)
IO.copyFile(original.toFile, destination.toFile)
}
}
Then you can invoke it like this:
copyTasties C:\\Dev\\sandbox\\a C:\\Dev\\sandbox\\b
If the original and destination are stable(for example they are directories inside the project) you can rewrite the task:
import java.nio.file.{Files, Paths}
import sbt._
val copyTastiesHardcoded = taskKey[Unit]("Copy .tasty files")
copyTastiesHardcoded := {
val baseDir = baseDirectory.value.toPath
val from = baseDir.resolve("a")
val to = baseDir.resolve("b")
Files
.walk(from)
.filter(Files.isRegularFile(_))
.filter(path => path.toString.endsWith("tasty"))
.forEach { original =>
val relative = from.relativize(original)
val destination = to.resolve(relative)
IO.copyFile(original.toFile, destination.toFile)
}
}
and invoke it without arguments
copyTastiesHardcoded

There are some useful utility methods in sbt that will help you find files and copy them around. Unfortunately the use a wild mixture of the traditional java.io.File and the more modern java.nio.file.Path, so you need to do conversions between them:
val copyFiles = taskKey[Unit]("copy files")
copyFiles := {
val inputDir = baseDirectory.value.toPath / "a"
val files: Seq[(Path, FileAttributes)] =
FileTreeView.default.list(inputDir.toGlob / RecursiveGlob / "*.tasty")
val outputDir = baseDirectory.value.toPath / "b"
IO.copy {
files.map { case (p, _) =>
p.toFile -> outputDir.resolve(inputDir.relativize(p)).toFile
}
}
}
Note that I'm using a glob here to find the files:
https://www.scala-sbt.org/1.x/docs/Globs.html

Related

creating function for loading conf file and store all properties in case class

I have one app.conf file like below:
some_variable_a = some_infohere_a
some_variable_b = some_infohere_b
Now I need to write scala function to load this app.conf file and create a scala case class to store all these properties. with try-catch and file checking conditions and corner cases.
I am very new to scala do not have much knowledge on this please provide me a correct way to do this.
Whatever I tried I am writing below:
import java.io.File
import com.typesafe.config.{ Config, ConfigFactory }
import com.typesafe.config._
import java.nio.file.Paths
private def ReadConfFile(path: String) = {
val fileTemp = new File(path)
if (fileTemp.exists) {
val confFile = Paths.get(Path).toFile
val config = ConfigFactory.parseFile(confFile)
val some_variable_a = config.getString("some_variable_a")
val some_variable_b = config.getString("some_variable_b")
}
}
Assuming that app.conf is in root folder of your application, this should be enough to access those variables from config file:
val config = ConfigFactory.load("app.conf")
val some_variable_a = config.getString("some_variable_a")
val some_variable_b = config.getString("some_variable_b")
In case you need to load it from the file using absolute path, you can do that in this way:
val myConfigFile = new File("/home/user/location/app.conf")
val config = ConfigFactory.parseFile(myConfigFile)
val some_variable_a = config.getString("some_variable_a")
val some_variable_b = config.getString("some_variable_b")
Or something similar as you did. In your code there is a typo I guess in Paths.get(Path).toFile. "Path" should be "path". If you If you don't have some variable Path, you should get compiling error for that. If that is not the problem, then check if you are providing correct path:
private def ReadConfFile(path: String) = {
val fileTemp = new File(path)
if (fileTemp.exists) {
val confFile = Paths.get(path).toFile
val config = ConfigFactory.parseFile(confFile)
val some_variable_a = config.getString("some_variable_a")
val some_variable_b = config.getString("some_variable_b")
}
}
ReadConfFile("/home/user/location/app.conf")

How to list the path of files inside Hdfs directory and subdirectory?

Could not figure out a way to list all files in directory and subdirectories.
Here is the code that I'm using which lists files in a specific directory but files if there is a subdorectory inside:
val conf = new Configuration()
val fs = FileSystem.get(new java.net.URI("hdfs://servername/"), conf)
val status = fs.listStatus(new Path("path/to/folder/"))
status.foreach { x => println(x.getPath.toString()) }
the above code lists all the files inside the directory but I need it to be recursive.
You could go for a recursion whenever you discover a new folder:
val hdfs = FileSystem.get(new Configuration())
def listFileNames(hdfsPath: String): List[String] = {
hdfs
.listStatus(new Path(hdfsPath))
.flatMap { status =>
// If it's a file:
if (status.isFile)
List(hdfsPath + "/" + status.getPath.getName)
// If it's a dir and we're in a recursive option:
else
listFileNames(hdfsPath + "/" + status.getPath.getName)
}
.toList
.sorted
}

how get element from List based on its name?

I have a path that contains subpath ,each subpath contains files
path="/data"
I implemented tow function to get csv files from each sub path
def getListOfSubDirectories(directoryName: String): Array[String] = {
(new File(directoryName))
.listFiles
.filter(_.isDirectory)
.map(_.getName)
}
def getListOfFiles(dir: String, extensions: List[String]): List[File] = {
val d = new File(dir)
d.listFiles.filter(_.isFile).toList.filter { file =>
extensions.exists(file.getName.endsWith(_))
}
}
each sub path contain 5 csv files : contextfile.csv,datafile.csv,datesfiles.csv,errors.csv,testfiles so my problem that i'll work with each file in a separate dataframe how I can get name of file for the right dataframe for example I want to get the name of files that concern context (i.e contextfile.csv). I worked like this but for each iteration the logic and the ranking in th List change
val dir=getListOfSubDirectories(path)
for (sup_path <- dir)
{ val Files = getListOfFiles(path + "//" + sup_path, List(".csv"))
val filename_context = Files(1).toString
val filename_datavalue = Files(0).toString
val filename_error = Files(3).toString
val filename_testresult = Files(4).toString
}
any help and thanks
I solve it just a simple filter
val filename_context = Files.filter(f =>f.getName.contains("context")).last.toString
val filename_datavalue = Files.filter(f =>f.getName.contains("data")).last.toString
val filename_error = Files.filter(f =>f.getName.contains("error")).last.toString
val filename_testresult = Files.filter(f =>f.getName.contains("test")).last.toString

Customise Freedesktop file with sbt-native-packager and JDKPackager plugin

I would like to customize the .desktop file created by the javapackager as part of the JDKPackager plugin of sbt-native-packager. It obviously uses a template:
[info] Using default package resource [Menu shortcut descriptor]
(add package/linux/Foo.desktop to the class path to customize)
In particular, I want to add the StartupWMClass entry that will be used by Gnome to unify all the windows opened by my application.
The javapackager refers to the target directory of the plugin, i.e. target/jdkpackager. This is created for example when the javafx-ant build-file is written. So we can piggyback here:
// rewrite the task so that after the ant build is created,
// we add package/linux/MyApp.desktop
writeAntBuild in JDKPackager := {
val res = (writeAntBuild in JDKPackager).value
val main = (mainClass in JDKPackager).value
.getOrElse(sys.error("No main class specified"))
val tgt = (target in JDKPackager).value
val n = (name in JDKPackager).value
val wm = main.replace('.', '-')
val desktop =
s"""[Desktop Entry]
|Name=APPLICATION_NAME
|Comment=APPLICATION_SUMMARY
|Exec=/opt/APPLICATION_FS_NAME/APPLICATION_LAUNCHER_FILENAME
|Icon=/opt/APPLICATION_FS_NAME/APPLICATION_LAUNCHER_FILENAME.png
|Terminal=false
|Type=Application
|Categories=DEPLOY_BUNDLE_CATEGORY
|DESKTOP_MIMES
|StartupWMClass=$wm
|""".stripMargin
IO.write(tgt / "package" / "linux" / s"$n.desktop", desktop)
res
}
As long as there is a corresponding setting in the Ant task XML, there is an alternative solution: just rewriting the XML generated by the plugin, via antBuildDefn, will suffice.
Here's an example of specifying a menu category for the .desktop file, via adding the category attribute :
antBuildDefn in JDKPackager := {
val origTask = (antBuildDefn in JDKPackager).value
val InfoLabel = "info"
val KeyRegex = s"$InfoLabel\\.(.+)".r
import scala.xml._
import scala.xml.transform._
val infoRewrite = new RewriteRule {
override def transform(n: Node) = n match {
case e: Elem if e.prefix == "fx" && e.label == InfoLabel =>
e % Attribute("", "category", "Office", Null)
case other => other
}
}
new RuleTransformer(infoRewrite)(origTask)
}

How do I get SBT staging directory at build time?

How do I get SBT staging directory at build time?
I want to do a tricky clone of a remote repo, and the stagingDirectory of SBT seems to be a nice fit.
How do I get the directory inside "Build.scala" ?
SBT source code:
http://www.scala-sbt.org/0.13.1/sxr/sbt/BuildPaths.scala.html#sbt.BuildPaths.stagingDirectory
=======
Underlying problem NOT directly relevant to the question. I wanted to use a subdirectory of a git dependency in SBT. SBT doesn't provide this out of the box so I wrote a simple wrapper:
object Git {
def clone(cloneFrom: String, branch: String, subdirectory: String) = {
val uniqueHash = Hash.halfHashString(cloneFrom + branch)
val cloneTo = file(sys.props("user.home")) / ".sbt" / "staging" / uniqueHash
val clonedDir = creates(cloneTo) {
Resolvers.run("git", "clone", cloneFrom, cloneTo.absolutePath)
Resolvers.run(Some(cloneTo), "git", "checkout", "-q", branch)
}
clonedDir / subdirectory
}
}
usage:
lazy val myDependency = Git.clone(cloneFrom = "git://...someproject.git", branch = "v2.4", subdirectory = "someModule")
Looking at the API from your link, there are two methods you can use getGlobalBase and getStagingDirectory, both take state.
import sbt._
import Keys._
import sbt.BuildPaths._
object MyBuild extends Build {
val outputStaging = taskKey[Unit]("Outputs staging")
lazy val root = project.in(file(".")).settings(
outputStaging := {
val s = state.value
println(getStagingDirectory(s, getGlobalBase(s)))
}
)
}
Edit
After your last comment, I think you're looking for a custom resolver. The custom resolver has an access to a ResolveInfo object, which has a property called staging.
For example this is how you could achieve what you're looking for (actually without accessing staging directly):
object MyBuild extends Build {
lazy val root = project.in(file(".")).dependsOn(RootProject(uri("dir+git://github.com/lpiepiora/test-repo.git#branch=master&dir=subdir")))
override def buildLoaders = BuildLoader.resolve(myCustomGitResolver) +: super.buildLoaders
def myCustomGitResolver(info: BuildLoader.ResolveInfo): Option[() => File] =
if(info.uri.getScheme != "dir+git") None
else {
import RichURI.fromURI
val uri = info.uri
val (branch, directory) = parseOutBranchNameAndDir(uri.getFragment)
val gitResolveInfo = new BuildLoader.ResolveInfo(
uri.copy(scheme = "git", fragment = branch), info.staging, info.config, info.state
)
println(uri.copy(scheme = "git", fragment = branch))
Resolvers.git(gitResolveInfo).map(fn => () => fn() / directory)
}
// just an ugly way to get the branch and the folder
// you may want something more sophisticated
private def parseOutBranchNameAndDir(fragment: String): (String, String) = {
val Array(branch, dir) = fragment.split('&')
(branch.split('=')(1), dir.split('=')(1))
}
}
The idea is that we delegate to the predefined git resolver, and we let it do it's work, after it's done, we'll return a subdirectory: fn() / directory.
This is an example and of course you could stick to your logic of getting the repository. The staging directory will be available to you in the resolver method.