How to read a file via sftp in scala - scala

I am looking for a simple way to read a file (and maybe a directory) via sftp protocol in scala.
My tries:
I've looked at the Alpakka library, which is a part of akka.
But this works with streams, which is a complex topic I am not familiar with and it seems to much effort for this.
Then there is spark-sftp: this needs scala spark, which would be a bit much just to load a file.
There is the jsch library for java that could do the job, but I could not bring it to work
I am looking for actual working code that uses a library and sftp instead of a plain scp, which I am forced to do. I've found that there are not many examples for this on the web, and the ones I have found are much more complex.

Here is a working example, using sshj:
import net.schmizz.sshj.SSHClient
import net.schmizz.sshj.sftp.SFTPClient
object Main extends App {
val hostname = "myServerName"
val username = "myUserName"
val password = "thePassword"
val destinationFile = "C:/Temp/Test.txt"
val sourceFile = "./Test.txt"
val ssh = new SSHClient()
ssh.addHostKeyVerifier("xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx")
ssh.connect(hostname)
ssh.authPassword(username, password)
val sftp: SFTPClient = ssh.newSFTPClient()
sftp.get(sourceFile, destinationFile)
sftp.close()
ssh.disconnect()
}
I tested this on scala version 2.13.4 with the following entries in build.sbt:
libraryDependencies += "com.hierynomus" % "sshj" % "0.31.0"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.3"
I would not recommend to actually use it this way. Some of these steps should be wrapped in a Try and then some error checking should be done if the file didn't exists or the connection failed and so on. I intentionally left that out for clarity.
I am not saying that this is the only or the right library for this task. It is just the first one that did work for me. Especially the addHostKeyVerifier method was very helpful in my case.
There are also other libraries like JSCH, jassh, scala-ssh and scala-ftp wich also could very well do the job.

Related

Can't get the banana-rdf to work with scalajs

From my understanding I should be able to use the banana-rdf library in my scalajs code? I followed the instructions on the website and added the following to my build.sbt:
val banana = (name: String) => "org.w3" %% name % "0.8.4" excludeAll (ExclusionRule(organization = "org.scala-stm"))
and added the following to my common settings:
resolvers += "bblfish-snapshots" at "http://bblfish.net/work/repo/releases"
libraryDependencies ++= Seq("banana", "banana-rdf", "banana-sesame").map(banana)
It all compiles fine until it gets to the point where it does the fast optimizing. Then I get the following error:
Referring to non-existent class org.w3.banana.sesame.Sesame$
I tried changing Seasame for Plantain but got the same outcome.
Am I missing something?
I was using the "%%" notation which is for a jvm module.
Changed it to "%%%" and it was able to find the correct library.
NOTE. I had to use Plantain as this is the only one currently compiled for scalajs

What can cause the Squeryl table object to be not found?

I am encountering a compile time error while attempting to get Squeryl example code running. The following code is based on the My Adventures in Coding blog post about connecting to SQLServer using Squeryl.
import org.squeryl.adapters.MSSQLServer
import org.squeryl.{ SessionFactory, Session}
import com.company.model.Consumer
class SandBox {
def tester() = {
val databaseConnectionUrl = "jdbc:jtds:sqlserver://myservername;DatabaseName=mydatabasename"
val databaseUsername = "userName"
val databasePassword = "password"
Class.forName("net.sourceforge.jtds.jdbc.Driver")
SessionFactory.concreteFactory = Some(()=>
Session.create(
java.sql.DriverManager.getConnection(databaseConnectionUrl, databaseUsername, databasePassword),
new MSSQLServer))
val consumers = table[Consumer]("Consumer")
}
}
I believe I have the build.sbt file configured correctly to import the Squeryl & JTDS libraries. When running SBT after adding the dependencies it appeared to download the libraries need.
libraryDependencies ++= List (
"org.squeryl" %% "squeryl" % "0.9.5-6",
"net.sourceforge.jtds" % "jtds" % "1.2.4",
Company.teamcityDepend("company-services-libs"),
Company.teamcityDepend("ssolibrary")
) ::: Company.teamcityConfDepend("company-services-common", "test,gatling")
I am certain that at least some of the dependencies were successfully installed. I base this on the fact that the SessionFactory code block compiles successfully. It is only the line that attempts to setup a map from the Consumer class to the Consumer SQLServer table.
val consumers = table[Consumer]("Consumer")
This line causes a compile time error to be thrown. The compile is not able to find the table object.
[info] Compiling 8 Scala sources to /Users/pbeacom/Company/integration/target/scala-2.10/classes...
[error] /Users/pbeacom/Company/integration/src/main/scala/com/company/code/SandBox.scala:25: not found: value table
[error] val consumers = table[Consumer]("Consumer")
The version of Scala in use is 2.10 and if the table line is commented the code compiles successfully. Use of the table object to accomplish data model mappings is nearly ubiquitous in the Squeryl examples I'm been researching online and no one else seems to have encountered a similar problem.
Shortly after posting this and reviewing it I finally noticed my problem. I was not being conscious enough of the heavy use of mixins in Scala. I failed to extend the Schema class. That's why table is unknown in the scope of the SandBox class. I was able to solve the problem with the following class definition:
class SandBox extends Schema {
def tester() = {
...

Clean solution for dropping into REPL console in the middle of program execution

Is there any working solution for dropping into REPL console with for Scala 2.10?
This is mainly for debugging purpose - I want to pause in the middle of execution, and have a REPL console where I can inspect values and test the program's logic using complex expressions within my program at the current state of execution. Those who have programmed in Ruby might know similar function: the binding.pry.
AFAIK, Scala 2.9 and under used to have breakIf but it has been removed from the later versions. Using ILoop seems to be the new way but introduced issues due to sbt not adding scala-library to the classpath.
Several solutions such as this and this seem to offer a good workaround but my point is there must be a solution where I don't have to spend hours or even days just to make the REPL working.
In short, there's a lot more boilerplate steps involved - this is in contrast with binding.pry which is just a line of code with no additional boilerplate.
I am not aware if there's an issue introduced in executing the program as an sbt task as opposed to if running the program executable directly, but for development purpose I am currently running and testing my program using sbt task.
You could easily reimplement the breakIf method in your code. I don't think there is much cleaner way of doing that.
First you have to add a scala compiler library to your build.sbt
libraryDependencies += "org.scala-lang" % "scala-compiler" % scalaVersion.value
Once that's done you can implement breakIf
import scala.reflect.ClassTag
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.{ILoop, NamedParam}
def breakIf[T](assertion: => Boolean, args: NamedParam*)(implicit tag: ClassTag[T]) = {
val repl = new ILoop()
repl.settings = new Settings()
repl.settings.embeddedDefaults[T]
repl.settings.Yreplsync.value = true
repl.in = repl.chooseReader(repl.settings)
repl.createInterpreter()
args.foreach(p => repl.bind(p.name, p.tpe, p.value))
repl.loop()
repl.closeInterpreter()
}
I think it's pretty straight forward, the only tricky part is that you have to set-up the classpath properly. You need to call embeddedDefaults with a class from your project (see my answer to another question).
You can use the new breakIf as follows:
val x = 10
breakIf[X](assertion = true, NamedParam("x", "Int", x))
Where X is just some of your classes.
I don't know if this answers your question, because it's hard to measure what is easy and what is hard.
Additionally just as a side note - if you want to use it for debugging purposes, why not use a debugger. I guess most of the debuggers can connect to a program, stop at a breakpoint and evaluate expressions in that context.
Edit
Seems like it doesn't work on current release of Scala 2.10, the working code seems to be:
import scala.reflect.ClassTag
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.{ILoop, NamedParam}
def breakIf[T](assertion: => Boolean, args: NamedParam*)(implicit tag: ClassTag[T]) = {
val repl = new ILoop() {
override protected def postInitialization(): Unit = {
addThunk(args.foreach(p => intp.bind(p)))
super.postInitialization()
}
}
val settings = new Settings()
settings.Yreplsync.value = true
settings.usejavacp.value = true
settings.embeddedDefaults[T]
args.foreach(repl.intp.rebind)
repl.process(settings)
}
and usage is like
val x = 10
breakIf[X](assertion = true, NamedParam("x", x))
I was looking at this recently and found Ammonite to be a sufficient solution for my needs.
Add Ammonite to your library dependencies:
libraryDependencies += "com.lihaoyi" % "ammonite" % "1.6.0" cross CrossVersion.full
Invoke Ammonite where you want to drop to the REPL shell:
ammonite.Main().run()
Note that you have to pass any variables you want bound inside of run, e.g. run("var1" -> var1). Have a look at their example - Instantiating Ammonite.

Play Iteratees: error for simple file iteration

I'm currently trying to wrap my head around the idea of Enumerators and Iteratees. I decided to start off by looking at Play 2.0's iteratee library, which I've added to my test project with the following lines in my build.sbt file. (I am using Scala 2.10) (docs here)
resolvers += "Typesafe repository" at
"http://repo.typesafe.com/typesafe/releases/"
libraryDependencies += "play" %% "play-iteratees" % "2.1.1"
My goal is to create an Enumerator over the bytes of a file, and eventually attach some parsing logic to it, but when I try what appears to be a simple thing, I get an exception.
My code looks like this:
val instr = getClass.getResourceAsStream(...)
val streamBytes = for {
chunk <- Enumerator fromStream instr
byte <- Enumerator enumerate chunk
} yield byte
val printer = Iteratee.foreach[Byte](println)
streamBytes.apply(printer)
What happens is that (what I assume is) all of the bytes in the file get printed, then I get an IllegalStateException saying that the "Promise already completed".
java.lang.IllegalStateException: Promise already completed.
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)
at scala.concurrent.Promise$class.failure(Promise.scala:107)
at scala.concurrent.impl.Promise$DefaultPromise.failure(Promise.scala:58)
at scala.concurrent.Future$$anonfun$flatMap$1.liftedTree3$1(Future.scala:283)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:277)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:274)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Since the stack trace doesn't point to anywhere in my code, and this is unfamiliar territory, I have no idea what's going wrong. Can anyone offer some insight or a solution to this problem?
See if this works for you. I was getting exceptions with your code too, but when I unwound your for-comp, things worked. I'm not 100% sure why because I thought the for-comp desugared to this code anyway, but I must be missing something:
val bytes = Enumerator fromStream instr flatMap (Enumerator enumerate _)
val printer = Iteratee.foreach[Byte](b => println(b))
bytes |>> printer

Is there somewhere a guide to SBT for non-Scala programmers?

Someday, I'd like to learn Scala. What I see about the language from people who like it is very encouraging.
Today, though, is not that day. Today, I'd just like to make some changes to my team's build file. Unfortunately, this build file was put together with SBT, and is nearly incomprehensible.
My main issue is that it appears to me that SBT introduces some huge collection of new operators that do things with strings and lists to create some sort of sbt object. For example, in sbt:
"args4j" % "args4j" % "2.0.12"
Apparently is actually defined; however, I can't even tell what type it is at the scala repl, since at the repl I get the sensible error:
scala> val tstcrap = "args4j" % "args4j" % "2.0.12"
<console>:6: error: value % is not a member of java.lang.String
val tstcrap = "args4j" % "args4j" % "2.0.12"
I get this error even after setting up the classpath to include the sbt-launch.jar file and doing import sbt._.
Likewise, I'm dealing with stuff like this:
val jarSources = (descendents(classesOutput ##, "*") ---
assemblyExclude(classesOutput ##))
What's that ## operator, what's that --- doing, and more importantly what is the type of this expression? Are all these new operators documented somewhere, and is there some way to get a scala repl that's using the same language as is used in the sbt build files?
Looking at this sbt file reminds me of trying to decipher perl without ever reading any of the relevant man pages. (Not a recommended activity)
Update: After looking at the links in the answer below, and looking at other questions and answers tagged sbt, I've come across the major piece of scala knowledge that I was missing: scala allows one to define implicit conversions that will be invoked before methods are resolved. In this case, sbt defines inside the ManagedProject trait, an implicit conversion from String to the private class sbt.GroupID, so that
"a" % "b"
Is really something like
(new GroupID("a")) % "b"
I imagine the resolution order and other rules around implicit conversions must get very complicated; it almost reminds me of the nightmares you can introduce in C++ with operator overloading when done through non-member functions.
Since an SBT build file is a full-fledged Scala source file and relies on some libraries provided by SBT itself, it's difficult to cover SBT well without relying on some familiarity with Scala. I'm not aware of such a guide.
For the specific questions you raise, I think these wiki pages will help:
% operator for strings: http://code.google.com/p/simple-build-tool/wiki/LibraryManagement
## and --- operators: http://code.google.com/p/simple-build-tool/wiki/Paths
If you want to get a Scala REPL running with the SBT libraries available, try this:
$ sbt console-project
Some other useful commands are listed at http://code.google.com/p/simple-build-tool/wiki/RunningSbt .
Update 2016 (5 years later).
This is not a complete guide, but the article "Sbt heiroglyphs and multi-projects explained" from Divan Visagie can help starting to use sbt.
Plus, the sbt documentation is quite complete nowadays, and covers multiple projects in a single build.
The '---' operator is described in the PathFinder (since the 0.2 version).