Is there a configuration file for Scala REPL / SBT Console? - scala

I've been trying to find some sort of a dotfile to put Scala REPL settings and custom function in.
In particular I'm interested in passing it flags like -Dscala.color (enables syntax highlighting), as well as overriding settings like result string truncation:
scala> :power
scala> vals.isettings.maxPrintString = 10000
It would be nice to have these settings apply to both the simple Scala REPL sessions as well as sbt console sessions.
Does such a central configuration place exist for Scala?

Maybe you can use a modernized Scala REPL:
https://lihaoyi.github.io/Ammonite/

Poor man's solution: Set yourself an alias
alias myScala='scala -Dscala.repl.axPrintString = 10000'

As mentioned here ~/.sbt/0.13/global.sbt is the global configuration file for sbt. You can change your global settings here, this probably not going to effect REPL but should do work with SBT Console

You mainly asked about property settings, this goes a little beyond that to consider loading a definitions file as well—and isn't much help for Windows—but I thought I'd share in case it's useful:
I've resorted to using a wrapper script saved as ~/bin/scala, to set config properties and load some utility functions:
#!/bin/sh
# The scala REPL doesn't have any config file, so this wrapper serves to set
# some property values and load an init file of utilities when run without
# arguments to enter REPL mode.
#
# If there are arguments, just assume we're running a .scala file in script
# mode, a class or jar, etc., and execute normally.
SCALA=${SCALA:-/usr/local/bin/scala}
if [ "$#" -eq 0 ] && [ -r ~/.config/scala/replinit.scala ]; then
exec "$SCALA" -i ~/.config/scala/replinit.scala -Dscala.color
else
exec "$SCALA" "$#"
fi
If you sometimes use Ammonite REPL, as another answer suggests, the utility definitions can be shared by loading them from ~/.ammonite/predef.scala:
try load.exec(ammonite.ops.home/".config"/'scala/"replinit.scala")
catch { case _: Exception => println("=== replrc not loaded! ===") }
I'm not sure about a way to load the init file for sbt console automatically, though—Seth Tisue's comment about the initialize setting is helpful for properties, but using a :load command in a value for initialCommands in console doesn't appear to work.

Related

fish shell functions not accessing user paths

I've recently started using fish, and I needed to use a jar file for google's bundletool.
As such, I needed to set up an alias/function for bundletool, and I chose a function since it seems more "fishy".
My function is simple:
function bundletool
java -jar bundletool-all-1.12.1.jar $argv
end
The bundletool jar itself lives at ~/.local/bin, which is on my fish user path:
lase#laser-razer /m/c/U/matth [1]> echo $fish_user_paths
/home/lase/.local/bin /usr/local/go/bin /home/lase/.nvm /home/lase/.cargo/bin /home/lase/.cargo
In a regular shell, I can execute java -jar bundletool-all-1.12.1.jar, and the command runs as expected. However, in the function, fish doesn't seem to know about my fish_user_paths, and reports it cannot find the jar file.
To remedy this, I had to update my function to specify the full path:
function bundletool
java -jar ~/.local/bin/bundletool-all-1.12.1.jar $argv
end
This works, but I feel like I'm doing something incorrectly. Should I be setting up my paths or functions in a different fashion?
Your function will run in the ordinary environment.
It will run with the current $PATH [0] and, more importantly to you, the current working directory.
What happens is this:
java -jar bundletool-all-1.12.1.jar
Will tell java to run the jar found at bundletool-all-1.12.1.jar.
Notably, fish just hands java the string "bundletool-all-1.12.1.jar", and java will then not look at $PATH. It does not care about $PATH. It simply looks at "bundletool-all-1.12.1.jar", and tries to open a file by that name.
And it will find that file, if it is in the current directory.
And that's the reason this worked for you when you executed it interactively - because you happened to be in that directory
And then you tried it with the function, but you tried it from a different directory, and so it didn't work.
This works, but I feel like I'm doing something incorrectly. Should I be setting up my paths or functions in a different fashion?
No, giving the full path to the file instead of relying on the working directory is the right thing to do.
[0]: $fish_user_paths is just a variable you set, that fish will then take care to add to $PATH. $PATH is the actual variable that fish and other tools (including any command fish starts, if it wants to) will use to find commands.

How to get .bashrc / .profile variables in Scala

I am unable to get access to the environmental variables in either ~/.bashrc or ~/.profile from Scala. How do I access the environmental variables from a Scala Process? Also I am unable to update the paths like this:
Process("myProgram", None, "PATH"-> ".:/path/to/myProgram").!!
However this works:
Process("/path/to/myProgram",None).!!
Works fine. However, when myProgram depends on some environmental variables being set this doesnt work anymore.
How do I change the PATH variable from a Scala program?
And even better, how can I get Scala to access the environmental variables from .bashrc or .profile. Currently none of those are available.
Thanks for your time and help
How do I access the environmental variables from a Scala Process?
The util.Properties object offers 3 different methods for inspecting the environmental variables that the Scala process/program has inherited. Here's an example:
util.Properties.envOrNone("LC_COLLATE")
//res0: Option[String] = Some(POSIX)
How do I change the PATH variable from a Scala program?
A running process is not allowed to alter its own environment, but it can launch a new process with a modified environment. There are a couple different ways to go about this.
One is to launch the shell of your choice and use shell syntax to make the modifications before invoking the target command.
import sys.process._
Seq("sh", "-c", "PATH=$PATH:$HOME/Progs myProg").!!
// a single arg:^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Or you can supply the environment mods as an argument to one of the many overloaded Process.apply() methods.
import scala.sys.process._
Process("./myProg"
,new java.io.File("/full/path/to")
,"PATH"->s"${Properties.envOrElse("PATH",".")}:/full/path/to"
).!!
...can I get Scala to access the environmental variables from .bashrc or .profile?
If your Scala program is launched from a shell with the proper environment then every process launched from your program should inherit the same. If, for whatever reason, your program has not inherited a fully equipped environment then the easiest thing to do is to launch a fully equipped shell to launch the target command.
import scala.sys.process._
Seq("sh", "-c" , ". $HOME/.bashrc && myProg").!!

Execute the scala script through spark-shell in silent mode

Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala", after the execution, I am getting into the scala interactive mode. I don't want to get into there.
I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.
spark-shell -i "file.scala"
after execution, I get into
scala>
I don't want to get into the scala> mode
Updating (October 2019) for a script that terminates
This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script). See this question with a good example.
It also needs a "silent mode", it is expected to not pollute the output.txt.
Suppose Spark v2.2+.
PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler... Please, "let's compile!" is not an answer here.
spark-shell -i file.scala keeps the interpreter open
in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.
If logging is requiered you can use something like this:
spark-shell < file.scala > test.log 2>&1 &
If you have limitations on editing file and you can't add System.exit(0), use:
echo :quit | scala-shell -i file.scala
UPD
If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here. Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.
The best way is definitively to compile your scala code to a jar and use spark-submit but if you're simply looking for a quick iteration loop, you can simply issue a :quit after parsing your scala code:
echo :quit | scala-shell -i yourfile.scala
Adding onto #rluta's answer. You can place the call to spark-shell command inside a shell script. Say the below in a shell script:
spark-shell < yourfile.scala
But this would require you to keep the lines of code within a line in case a statement is written on different lines.
OR
echo :quit | spark-shell -i yourfile.scala
This should

supervisor program:x command expansion of environment variables $(ENV_VAR)s?

I would like to put configuration (in this case, site name) into supervisor
environment variables, for expansion in program:x command arguments. Is this supported? The documentation's wording would seem to indicate yes.
The following syntax is not working for me on supervisor-3.0 (excerpt of config file):
[supervisord]
environment = SITE="mysite"
[program:service_name]
command=/path/to/myprog/myservice /data/myprog/%(ENV_SITE)s/%(ENV_SITE)s.db %(program_name)s_%(process_num)03d
process_name=%(program_name)s_%(process_num)03d
numprocs=5
numprocs_start=1
Raises the following error:
sudo supervisord -c supervisord.conf
Error: Format string
'/path/to/myprog/myservice /data/myprog/%(ENV_SITE)s/%(ENV_SITE)s.db %(program_name)s_%(process_num)03d'
for 'command' contains names which cannot be expanded
Reading the documentation, I expected environment variables to be available for
expansion in program:x command as %(ENV_VAR)s:
http://supervisord.org/configuration.html#program-x-section-values
command:
"String expressions are evaluated against a dictionary containing the keys
group_name, host_node_name, process_num, program_name, here (the directory of
the supervisord config file), and all supervisord's environment variables
prefixed with ENV_."
Introduced: 3.0
Related:
There are open pull requests to enable expansion in additional section values:
https://github.com/Supervisor/supervisor/issues?labels=expansions&page=1&state=open
A search of goole (or SO) returns no examples of attempts to use %(ENV_VAR)s
expansion in the command section value:
https://www.google.com/search?q=supervisord+environment+expansion+in+command
I agree supervisor is not clear about this ( to me at least ).
I've found the easiest solution to execute /bin/bash -c.
In your case it would be:
command=/bin/bash -c"/path/to/myprog/myservice /data/myprog/${SITE}/${SITE}.db ..."
What do you think?
I've found inspiration here: http://blog.trifork.com/2014/03/11/using-supervisor-with-docker-to-manage-processes-supporting-image-inheritance/
You are doing it right; however, the ENV defined in your supervisord section doesn't get made available to the processes for whatever reason during configuration loading. If you start supervisord like this:
SITE=mysite supervisord
It will run correctly and expand that variable. I don't know why supervisord has issues adding to the environment and making it available to the subprocesses' config expansion. I think the environment variable is available inside the subprocess, but not when expanding variables in the subprocess config declaration.

can I attach functions to the coffee console before it starts?

Let's say I want to have a pp function:
coffee> pp = (obj) -> JSON.stringify(obj)
[Function]
coffee> pp({cat: "fancy"})
'{"cat":"fancy"}'
coffee>
Is there a way that I can have that function be available immediately when the console loads? I'm looking at coffee -r "utils.coffee", but don't see any way to put that required library into an object that's available at the command line. It looks like I might be able to alter repl.js, but that seems like a bad idea.
You can put those functions into the global variable (which is kinda Node's version of window) and then use the -r option.
# utils.coffee
global.pp = (obj) -> JSON.stringify(obj)
And then run, on the same directly utils.coffee of:
coffee -r ./utils
It should start a CoffeeScript REPL and have the pp function available as a global variable:
coffee> pp ohmy: 'neat'
'{"ohmy":"neat"}'
Update: it seems the -r command-line option was removed in CoffeeScript's master. It probably wasn't meant to be used this way :(
Update 2: There's another way to do this! And it doesn't rely on any command-specific parameter:
{ echo "require './utils'"; cat; } | coffee
It will, however, not work 100% like the Coffee REPL. The arrow keys for example don't seem to work.
Edit (jc): Using this method allows you to make an unload for node, which is handy:
# utils.coffee
global.unload = (moduleName) ->
cacheName = require.resolve(moduleName)
delete require.cache[cacheName]
$ coffee -r ~/Dev/utils.coffee
coffee> unload
[Function]
Update 3: Another possibility is to "create your own REPL". Not really reimplementing anything. Based on this hacky solution, you could do something like:
#! /usr/bin/env coffee
# REPL functions.
global.pp = (obj) -> JSON.stringify(obj)
# Start the REPL.
require 'coffee-script/lib/coffee-script/repl'
And then use that script as your new REPL. It will work exactly like the normal Coffee REPL plus the new functions (no problems with arrow keys nor TAB completion :)
BTW, i think you'll need to have CoffeeScript installed without the -g option on npm for that to work.
It is a very hacky solution though. It relies on the internal CoffeeScript implementation file structure and its functionality, and that could change at any moment (in fact, i'm aware that there has been some work done in a new revamped Coffee REPL based on Node's one... i hope that functionality gets exposed to be used programmatically, so these kind of hacks are not hacks any more).
Have a look at #Daniel's (Daniel Taylor's) excellent nesh, an enhanced node shell that also speaks CoffeeScript: http://danielgtaylor.github.io/nesh/
Here's how you would load your pp function into a CoffeeScript REPL:
nesh -c -e 'pp = (obj) -> JSON.stringify(obj)'
Similarly, if you wanted to load a file, say ~/.nesh_profile.coffee (analogous to loading a shell profile), on startup, use:
nesh -c -e ~/.nesh_profile.coffee
Small caveat: The REPL language - JavaScript v. CoffeeScript - and the language of the code you're loading must be the same. Edit: The only exception is that you can always load JavaScript via a file with extension .js, even when starting a CoffeeScript REPL. (By contrast, you can't specify a JS string when starting a CoffeeScript REPL).
To automate loading of your 'nesh profile' into a CoffeeScript REPL you could define an alias in your shell profile; e.g.:
alias neshc='nesh -c -e ~/.nesh_profile.coffee'
Update: Here's how you can use the very same 'nesh profile' with a JavaScript REPL; compilation to JavaScript is performed on the fly using process substitution:
alias neshj='nesh -e <(coffee -bp ~/.nesh_profile.coffee)'
nesh is available as an npm package, so you can easily install it as follows:
sudo npm install nesh -g