I have to do this:
scala -version 2>&1 | sed 's/.*version \([0-9]*\.[0-9]*\).*/\1/'
Instead of this:
scala -version | sed 's/.*version \([0-9]*\.[0-9]*\).*/\1/'
so I am wondering why does scala -version (sometimes maybe ...) put its result on stderr?
Actually, in Unix stderr is not used for errors only.
It's more like stdin and stdout are using for the default piping streams (cmd1 | cmd2 | cmd3) and sdterr is everything that the command creators thought that should not go into this default pipeline, and instead should be shown to the user who build it. It includes errors, but sometimes also information about progress, configuration, diagnostics, etc.
Let's see some example of that:
echo "2 + 2 * 2" | scala | grep "res"
This would treat scala as something that takes source as input and prints the results of evaluation on output. But it still prints some stuff on stderr, and when you run it you'll see
Type in expressions for evaluation. Or try :help.
Jul 08, 2022 9:32:44 AM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
scala> val res0: Int = 6
On a first glance it is redundant but when you look closely it tells some interesting things like WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information) which is nice to see as you are debugging your script, and not go into the stream where you are only expecting to see the results of the execution.
What would happen if you added -version after scala?
> echo "2 + 2 * 2" | scala -version | grep "res"
Scala code runner version 2.13.8 -- Copyright 2002-2021, LAMP/EPFL and Lightbend, Inc.
Script still runs - it's shell so you can glue a lot of things together whether it makes sense or not, so it's expected. But this script doesn't do anything reasonable - there is no code processing output from scala -version and if version landed on stdin you'd see empty result, not knowing what happened. But since it landed on stderr you see what happened, and can fix the script.
And if you actually wanted to process scala -version output, it's just one 2>&1 away as you noticed.
There are also other things to consider like mentioned in comments: consistency with what java -version does, no promise to print such information to stdin. Also, it might have been made this way to be internally consistent: scala might use stdin for code input and stdout for code execution exclusively, and anything else is wired to some global logger printing on stderr. Finally, while there are some conventions what should and should not go into stderr, none of that is set in stone, so you should not come with predefined expectations. Authors will do what they find the most pragmatic.
Related
I am developing a Qt application in Python. It uses a resource file, which needs to be compiled. I am using autotools to manage compilation and installation of my projects.
Now, in order for the resource file to be usable by the application, it needs to be compiled with a certain version of the compilation program (pyrcc). I can get the version by putting the output of pyrcc -version in a variable in configure.ac. But then, I don't know how to check whether the string pyrcc5 is present in the output. If it is not present, I want to tell the user that his PyRCC programm has the wrong version, and abort configure.
Additionally, I would like to avoid the need of an extra variable for the program output, but instead do it like this (Pseudo code):
if "pyrcc5" not in output of "pyrcc -version":
say "pyrcc has wrong version"
exit 1
How can I do this ?
When writing a configure.ac for Autoconf, always remember that you are basically writing a shell script. Autoconf provides a host of macros that afford you a lot of leverage, but you can usually at least get an idea about basic "How can I do X in Autoconf?" questions by asking instead "How would I do X in a portable shell script?"
In particular, for ...
I would like to avoid the need of an extra variable for the program
output, but instead do it like this (Pseudo code):
if "pyrcc5" not in output of "pyrcc -version":
say "pyrcc has wrong version"
exit 1
... the usual tool for a portable shell script to use for such a task is grep, and, happily, the easiest way to apply it to the task does not require an intermediate variable. For example, this implements exactly your pseudocode (without emitting any extraneous messaging to the console):
if ! pyrcc -version | grep pyrcc5 >/dev/null 2>/dev/null; then
echo "pyrcc has wrong version"
exit 1
fi
That pipes the output of pyrcc -version into grep, and relies on the fact that grep exits with a success status if and only if it finds any matches.
You could, in fact, put exactly that in your configure.ac, but it would be more idiomatic to
Use the usual Autoconf mechanisms to locate pyrcc and grep, and to use the versions discovered that way;
Use the Autoconf AS_IF macro to write the if construct, instead of writing it literally;
Use standard Autoconf mechanisms for emitting a "checking..." message and reporting on its result; and
Use the standard Autoconf mechanism for outputting a failure message and terminating.
Of course, all of that makes the above considerably more complex, but also more flexible and portable. It might look like this:
AC_ARG_VAR([PYRCC], [The name or full path of pyrcc. Version 5 is required.])
# ...
AC_PROG_GREP
AC_CHECK_PROGS([PYRCC], [pyrcc5 pyrcc], [])
AS_IF([test "x${PYRCC}" = x],
[AC_MSG_ERROR([Required program pyrcc was not found])])
# ...
AC_MSG_CHECKING([whether ${PYRCC} has an appropriate version])
AS_IF([! pyrcc -version | grep pyrcc5 >/dev/null 2>/dev/null], [
AC_MSG_RESULT([no])
AC_MSG_ERROR([pyrcc version 5 is required, but ${PYRCC} is a different version])
], [
AC_MSG_RESULT([yes])
])
In addition to portability and conventional Autoconf progress messaging, that also gets the builder a way to specify a particular pyrcc executable to configure (by setting variable PYRCC in its environment), documents that in configure's help text, and exports PYRCC as a make variable.
Oh, and I snuck in a check for pyrcc under the name pyrcc5, too, though I don't know whether that's useful in practice.
The final result no longer looks much like the shell script fragment I offered first, I grant. But again, the pure shell script fragment could be used as is, and also, the fully Autoconfiscated version is derived directly from the pure script.
Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala", after the execution, I am getting into the scala interactive mode. I don't want to get into there.
I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.
spark-shell -i "file.scala"
after execution, I get into
scala>
I don't want to get into the scala> mode
Updating (October 2019) for a script that terminates
This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script). See this question with a good example.
It also needs a "silent mode", it is expected to not pollute the output.txt.
Suppose Spark v2.2+.
PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler... Please, "let's compile!" is not an answer here.
spark-shell -i file.scala keeps the interpreter open
in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.
If logging is requiered you can use something like this:
spark-shell < file.scala > test.log 2>&1 &
If you have limitations on editing file and you can't add System.exit(0), use:
echo :quit | scala-shell -i file.scala
UPD
If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here. Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.
The best way is definitively to compile your scala code to a jar and use spark-submit but if you're simply looking for a quick iteration loop, you can simply issue a :quit after parsing your scala code:
echo :quit | scala-shell -i yourfile.scala
Adding onto #rluta's answer. You can place the call to spark-shell command inside a shell script. Say the below in a shell script:
spark-shell < yourfile.scala
But this would require you to keep the lines of code within a line in case a statement is written on different lines.
OR
echo :quit | spark-shell -i yourfile.scala
This should
First, some background. I'm trying to use cvs annotate within a Perl script invoked from a ksh command line to find out who's using bad hex constants in their source code and dole out justice appropriately. I'd like to use stdout for this program strictly for my own structured output, so I can pipe it to a file or to other programs. But, every call to cvs annotate results in a diagnostic message being printed to the console. It takes the form of:
Annotations for <filename>
***********
It's mucking up my stdout! I played around with annotate on the command line, and I figured that these diagnostic messages were coming from stderr, because running this command directly in ksh:
cvs annotate <filename> 1>yay.txt 2>boo.txt
correctly puts the desired annotated output into yay.txt and the diagnostics into boo.txt. However, things get weirder when I try to run this from within a perl script using backticks.
$muhstring = `cvs annotate $filename 2>boo.txt`;
The desired annotated output does appear in $muhstring, but the diagnostics still get printed to the command line, and boo.txt is created as an empty file. There's an admittedly old post on perlmonks.org that says this syntax should work as written. So, what's the deal? Is this a CVS quirk, have I misread the post, or have things changed since 1999? Or is it something else?
Edit: So, it is not, in fact, mucking up stdout. I tried this in ksh:
perl findbadhexowners.pl badhex.txt > out.txt
and the diagnostic messages still printed to the console, but out.txt only contains the annotations. This is fine for my purposes, and I can continue with other code.
That said, it seems like Perl and cvs specifically are interacting a little strangely, especially considering that redirection in ksh works fine and redirection in Perl for other commands like cd works fine - for example, in Perl
`cd nonexistentDir 1>stdout.txt 2>stderr.txt`;
gives the expected output in both stdout.txt and stderr.txt. It's only using cvs annotate from within Perl that produces this problem.
What is the name of startup script for the scala REPL. For example something along the lines of the following:
~/.scalarc
You could try something like:
$ alias scala='scala -i ~/.scalarc '
Note the trailing space—if you omit it your alias will disregard parameters.
Further use of 'scala' (once the alias is defined) will work as expected for REPL. If you use the alias for launching compiled programs, '~/.scalarc' will simply be ignored.
edit: It seems using '-i' in this way causes a significant slowdown.
The following, though somewhat convoluted (warning: bashism ahead), works faster:
$ scala -i <( cat ~/.scalarc foo.scala)
This concatenates your code (e.g. 'foo.scala') with '.scalarc' and evaluates everything on startup, leaving you at the REPL.
I don't think it's a satisfactory solution, but worth mentioning.
during my work here I collided with a somewhat peculiar problem. It's possible that there is a highly simple explanation for this behaviour, but to me it just doesn't make much sense.
Here's the situation:
I wrote a batch file "test.bat" that, right now, looks like this:
echo 1
scala myProgram
echo 2
When I open the command prompt in the according directory and run test.bat, it starts by echoing 1, then runs myProgram (which also has certain outputs that appear in the console, so the scala program myProgram works properly) - and then stops. 2 does not appear in the console and the console waits for me to input another command.
Why this behaviour? Is is a malfunction of the console? Or of the scala command? Or not a malfunction at all and it is actually meant to behave that way?
What I was actually trying to do is redirecting the output of "scala myProgram" to a file (which works well) and rename this file after the scala program has terminated, so my batch file originally looked somewhat like this:
scala myProgram > log.txt 2>&1
ren "log.txt" "log2.txt"
And I was confused about the fact that "log2.txt" was never created.
Your answers are greatly appreciated, thank you.
Adding -nc to scala command worked for me:
$ scala -nc /tmp/2.scala
Hello world
So I guess, the issue has something to do with the compilation daemon
-nc no compilation daemon: do not use the fsc offline compiler
Could you try that?