I'm trying to do sbt flywayMigrate -Denvi=foo but the system property envi is not being set. Pointers for debugging is greatly appreciated as I haven't been successful in identifying the cause of this issue for hours now. No question in SO or anywhere else have had this issue so far.
In build.sbt, this will be used as a variable.
lazy val envi = sys.props.getOrElse("envi", "default")
Using sys.env.get("ENVI") instead is currently not an option due to shared/team repo considerations.
sbt console -Denvi=foo
scala> sys.props.get("envi")
res0: Option[String] = None
scala> sys.props.getOrElse("envi", "default")
res1: Option[String] = default
scala, sbt installed using brew
You have to put the environment variable before the command:
sbt -Denvi=foo console
otherwise it will be passed as an argument to the main class instead of to the JVM.
Alternatively you can set the environment in the JAVA_OPTS variable before starting sbt:
export JAVA_OPTS="-Denvi=foo"
sbt console
scala> sys.props.getOrElse("envi", "default")
res0: String = foo
Related
In the book, Programming in Scala 5th Edition, the author says the following for two classes:
Neither ChecksumAccumulator.scala nor Summer.scala are scripts, because they end in a definition. A script, by contrast, must end in a result expression.
The ChecksumAccumulator.scala is as follows:
import scala.collection.mutable
class CheckSumAccumulator:
private var sum = 0
def add(b: Byte): Unit = sum += b
def checksum(): Int = ~(sum & 0XFF) + 1
object CheckSumAccumulator:
private val cache = mutable.Map.empty[String, Int]
def calculate(s: String): Int =
if cache.contains(s) then
cache(s)
else
val acc = new CheckSumAccumulator
for c<-s do
acc.add((c >> 8).toByte)
acc.add(c.toByte)
val cs = acc.checksum()
cache += (s -> cs)
cs
whereas the Summer.scala is as follows:
import CheckSumAccumulator.calculate
object Summer:
def main(args: Array[String]): Unit =
for arg <- args do
println(arg + ": " + calculate(arg))
But when I run the Summer.scala file, I get a different error than what mentioned by the author:
➜ learning-scala git:(main) ./scala3-3.0.0-RC3/bin/scala Summer.scala
-- [E006] Not Found Error: /Users/avirals/dev/learning-scala/Summer.scala:1:7 --
1 |import CheckSumAccumulator.calculate
| ^^^^^^^^^^^^^^^^^^^
| Not found: CheckSumAccumulator
longer explanation available when compiling with `-explain`
1 error found
Error: Errors encountered during compilation
➜ learning-scala git:(main)
The author mentioned that the error would be around not having a result expression.
I also tried to compile CheckSumAccumulator only and then run Summer.scala as a script without compiling it:
➜ learning-scala git:(main) ./scala3-3.0.0-RC3/bin/scalac CheckSumAccumulator.scala
➜ learning-scala git:(main) ✗ ./scala3-3.0.0-RC3/bin/scala Summer.scala
<No output, given no input>
➜ learning-scala git:(main) ✗ ./scala3-3.0.0-RC3/bin/scala Summer.scala Summer of love
Summer: -121
of: -213
love: -182
It works.
Obviously, when I compile both, and then run Summer.scala, it works as expected. However, the differentiation of Summer.scala as a script vs normal file is unclear to me.
Let's start top-down...
The most regular way to compile Scala is to use a build tool like SBT/Maven/Mill/Gradle/etc. This build tool will help with a few things: downloading dependencies/libraries, downloading Scala compiler (optional), setting up CLASS_PATH and most importantly running scalac compiler and passing all flags to it. Additionally it can package compiled class files into JARs and other formats and do much more. Most relevant part is CP and compilation flags.
If you strip off the build tool you can compile your project by manually invoking scalac with all required arguments and making sure your working directory matches package structure, i.e. you are in the right directory. This can be tedious because you need to download all libraries manually and make sure they are on the class path.
So far build tool and manual compiler invocation are very similar to what you can also do in Java.
If you want to have an ah-hoc way of running some Scala code there are 2 options. scala let's you run scripts or REPL by simply compiling your uncompiled code before it executes it.
However, there are some caveats. Essentially REPL and shell scripts are the same - Scala wraps your code in some anonymous object and then runs it. This way you can write any expression without having to follow convention of using main function or App trait (which provides main). It will compile the script you are trying to run but will have no idea about imported classes. You can either compile them beforehand or make a large script that contains all code. Of course if it starts getting too large it's time to make a proper project.
So in a sense there is no such thing as script vs normal file because they both contain Scala code. The file you are running with scala is a script if it's an uncompiled code XXX.scala and "normal" compiled class XXX.class otherwise. If you ignore object wrapping I've mentioned above the rest is the same just different steps to compile and run them.
Here is the traditional 2.xxx scala runner code snippet with all possible options:
def runTarget(): Option[Throwable] = howToRun match {
case AsObject =>
ObjectRunner.runAndCatch(settings.classpathURLs, thingToRun, command.arguments)
case AsScript if isE =>
ScriptRunner(settings).runScriptText(combinedCode, thingToRun +: command.arguments)
case AsScript =>
ScriptRunner(settings).runScript(thingToRun, command.arguments)
case AsJar =>
JarRunner.runJar(settings, thingToRun, command.arguments)
case Error =>
None
case _ =>
// We start the repl when no arguments are given.
if (settings.Wconf.isDefault && settings.lint.isDefault) {
// If user is agnostic about -Wconf and -Xlint, enable -deprecation and -feature
settings.deprecation.value = true
settings.feature.value = true
}
val config = ShellConfig(settings)
new ILoop(config).run(settings)
None
}
This is what's getting invoked when you run scala.
In Dotty/Scala3 the idea is similar but split into multiple classes and classpath logic might be different: REPL, Script runner. Script runner invokes repl.
I am running Spark-Shell with Scala and I want to set an environment variable to load data into Google bigQuery. The environment variable is GOOGLE_APPLICATION_CREDENTIALS and it contains /path/to/service/account.json
In python environment I can easily do,
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "path/to/service/account.json"
However, I cannot do this in Scala. I can print out the system environment variables using,
scala> sys.env
or
scala> System.getenv()
which returns me a map of String Key,Value pairs. However,
scala> System.getenv("GOOGLE_APPLICATION_CREDENTIALS") = "path/to/service/account.json"
returns an error
<console>:26: error: value update is not a member of java.util.Map[String,String]
I found a work around for this problem, though I dont think its the best practice. Here is the 2 step solution for this -
From terminal/cmd, first create the environment variable -
export GOOGLE_APPLICATION_CREDENTIALS=path/to/service/account.json
From the same terminal window, open spark-shell and run -
System.getenv("GOOGLE_APPLICATION_CREDENTIALS")
Friends,
I'm trying to set the environment variable "asdf" in my Scala shell, as described here
These are my commands:
scala> import scala.sys.process.Process
import scala.sys.process.Process
scala> Process(Seq("bash", "-c", "echo $asdf"), None, "asdf" -> "Hello, world!").!
Hello, world!
res18: Int = 0
But when i try to read the environment variable back:
scala> sys.env.get("asdf")
res19: Option[String] = None
The output says "None". How do i properly set my environment variable in the current session?
PS - Please do not downvote this; i'm trying really hard but unable to get past my issue
It has nothing to do with Scala, you are just misunderstanding the situation. The map at the end of the line
scala> Process(Seq("bash", "-c", "echo $asdf"), None, "asdf" -> "Hello, world!").!
doesn't change the environment of of this process, the one you are typing into; it changes the environment of the child process that the Process() function creates.
It is not permitted for a Scala/Java process to modify its own environment. You can use the scala.util.Properties object to inspect environmental variables and properties. (Docs are here.) The properties can be added/removed/changed but the environmentals cannot.
I have installed Tachyon and Spark according to instructions:
http://tachyon-project.org/documentation/Running-Spark-on-Tachyon.html
However, as a newbie I have no idea how to put file "X" into Tachyon File System as they said:
$ ./spark-shell
$ val s = sc.textFile("tachyon-ft://stanbyHost:19998/X")
$ s.count()
$ s.saveAsTextFile("tachyon-ft://activeHost:19998/Y")
What I did was to point to an existing file (that I find through the management UI):
scala> val s = sc.textFile("tachyon-ft://localhost:19998/root/default_tests_files/BasicFile_THROUGH")
s: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:21
When I run count, I got this below error:
scala> s.count()
java.lang.NullPointerException: connectionString cannot be null
I assume my path was wrong. So two questions:
How to copy a file into Tachyon?
What is the proper path for its FS?
Sorry, very very newbie !!
UPDATE 1
I am not sure if tachyon-ft://localhost:19998/root/default_tests_files/BasicFile_THROUGH is correct path. I cannot get it either via the browser or wget
This is what I saw in the file system browser
I found out the issue. I didn't do this
sc.hadoopConfiguration.set("fs.tachyon.impl", "tachyon.hadoop.TFS")
After I went through this exercise http://ampcamp.berkeley.edu/5/exercises/tachyon.html#run-spark-on-tachyon, I found out the proper path is this:
val file = sc.textFile("tachyon://localhost:19998/LICENSE")
So my setup was fine afterall. The documentation here http://tachyon-project.org/documentation/Running-Spark-on-Tachyon.html was causing me a lot of confusion.
I want use Scala like Python, so I install REPL in Sublime Text(Os is win8)
Everytime in REPL, I have to
scala> :load <my file>
, so I think it's inconvenient.
And I can't change
scala> :settings -d <路径名>
in Chinese directory.
I'm confused whether I can't change Scala script's directory with non-english language.
Thanks a lot!
If you use sbt then you can define initial commands when you launch the console.
yourproject/build.sbt:
// build.sbt
name := "initial-commands-example"
initialCommands := "import Foo._"
yourproject/script.scala:
// script.scala
object Foo {
def hello(name: String) = s"hello $name"
val msg = hello("world")
}
Inside yourproject, run sbt console, and you will have everything in Foo available inside that repl. See sbt initialCommands docs for more information.