during my work here I collided with a somewhat peculiar problem. It's possible that there is a highly simple explanation for this behaviour, but to me it just doesn't make much sense.
Here's the situation:
I wrote a batch file "test.bat" that, right now, looks like this:
echo 1
scala myProgram
echo 2
When I open the command prompt in the according directory and run test.bat, it starts by echoing 1, then runs myProgram (which also has certain outputs that appear in the console, so the scala program myProgram works properly) - and then stops. 2 does not appear in the console and the console waits for me to input another command.
Why this behaviour? Is is a malfunction of the console? Or of the scala command? Or not a malfunction at all and it is actually meant to behave that way?
What I was actually trying to do is redirecting the output of "scala myProgram" to a file (which works well) and rename this file after the scala program has terminated, so my batch file originally looked somewhat like this:
scala myProgram > log.txt 2>&1
ren "log.txt" "log2.txt"
And I was confused about the fact that "log2.txt" was never created.
Your answers are greatly appreciated, thank you.
Adding -nc to scala command worked for me:
$ scala -nc /tmp/2.scala
Hello world
So I guess, the issue has something to do with the compilation daemon
-nc no compilation daemon: do not use the fsc offline compiler
Could you try that?
Related
Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala", after the execution, I am getting into the scala interactive mode. I don't want to get into there.
I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.
spark-shell -i "file.scala"
after execution, I get into
scala>
I don't want to get into the scala> mode
Updating (October 2019) for a script that terminates
This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script). See this question with a good example.
It also needs a "silent mode", it is expected to not pollute the output.txt.
Suppose Spark v2.2+.
PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler... Please, "let's compile!" is not an answer here.
spark-shell -i file.scala keeps the interpreter open
in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.
If logging is requiered you can use something like this:
spark-shell < file.scala > test.log 2>&1 &
If you have limitations on editing file and you can't add System.exit(0), use:
echo :quit | scala-shell -i file.scala
UPD
If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here. Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.
The best way is definitively to compile your scala code to a jar and use spark-submit but if you're simply looking for a quick iteration loop, you can simply issue a :quit after parsing your scala code:
echo :quit | scala-shell -i yourfile.scala
Adding onto #rluta's answer. You can place the call to spark-shell command inside a shell script. Say the below in a shell script:
spark-shell < yourfile.scala
But this would require you to keep the lines of code within a line in case a statement is written on different lines.
OR
echo :quit | spark-shell -i yourfile.scala
This should
First, some background. I'm trying to use cvs annotate within a Perl script invoked from a ksh command line to find out who's using bad hex constants in their source code and dole out justice appropriately. I'd like to use stdout for this program strictly for my own structured output, so I can pipe it to a file or to other programs. But, every call to cvs annotate results in a diagnostic message being printed to the console. It takes the form of:
Annotations for <filename>
***********
It's mucking up my stdout! I played around with annotate on the command line, and I figured that these diagnostic messages were coming from stderr, because running this command directly in ksh:
cvs annotate <filename> 1>yay.txt 2>boo.txt
correctly puts the desired annotated output into yay.txt and the diagnostics into boo.txt. However, things get weirder when I try to run this from within a perl script using backticks.
$muhstring = `cvs annotate $filename 2>boo.txt`;
The desired annotated output does appear in $muhstring, but the diagnostics still get printed to the command line, and boo.txt is created as an empty file. There's an admittedly old post on perlmonks.org that says this syntax should work as written. So, what's the deal? Is this a CVS quirk, have I misread the post, or have things changed since 1999? Or is it something else?
Edit: So, it is not, in fact, mucking up stdout. I tried this in ksh:
perl findbadhexowners.pl badhex.txt > out.txt
and the diagnostic messages still printed to the console, but out.txt only contains the annotations. This is fine for my purposes, and I can continue with other code.
That said, it seems like Perl and cvs specifically are interacting a little strangely, especially considering that redirection in ksh works fine and redirection in Perl for other commands like cd works fine - for example, in Perl
`cd nonexistentDir 1>stdout.txt 2>stderr.txt`;
gives the expected output in both stdout.txt and stderr.txt. It's only using cvs annotate from within Perl that produces this problem.
I'm sure this is an easy fix, but I need to use "script" (and not collect standard in/out/error) for my project. I'm somewhat new to Perl, so please bear with me.
I have a Perl script that works fine. When I run it I generally type script > filename before I run Perl.
$script > file.log
bash-3.2$ perl foobar.pl
This runs fine, and when I'm done I type exit or control D to stop the script and save the file. All I'd like to do is incorporate the script command in Perl and then automatically capture the file when the program stops running (12-16 hours). The problem I have is that is I call in system("script > file.log"); and them call system("perl foobar.pl"); it hangs at the bash-3.2$ prompt. The only way to get Perl to work is control D or exit, stopping the script function.
Anyone have any idea how to fix this? While it's easy to start with script before invoking Perl, if I'm a mole and forget, I have to rerun the program (which takes a long time).
Have you considered using system("script -c 'perl foobar.pl' file.log")?
I am using mr on Windows and it allows running arbitrary commands before/after any repository action. As far as I can see this is done simply by invoking perl's system function. However something seems very wrong with my setup: when making mr run the following batch file, located in d:
#echo off
copy /Y foo.bat bar.bat
I get errors on the most basic windows commands:
d:/foo.bat: line 1: #echo: command not found
d:/foo.bat: line 2: copy: command not found
To make sure mr isn't the problem, I ran perl -e 'system( "d:/foo.bat" )' but the output is the same.
Using xcopy instead of copy, it seems the xcopy command is found since the output is now
d:/foo.bat: line 1: #echo: command not found
Invalid number of parameters
However I have no idea what could be wrong with the parameters. I figured maybe the problem is the batch file hasn't full access to the standard command environment so I tried running it explicitly via perl -e 'system( "cmd /c d:\foo.bat" )' but that just starts cmd and does not run the command (I have to exit the command line to get back to the one where I was).
What is wrong here? A detailed explanation would be great. Also, how do I solve this? I prefer a solution that leaves the batch file as is.
The echo directive is executed directly by the running command-prompt instance.
But perl is launching a new process with your command. You need to run your script within a cmd instance, for those commands to work.
Your cmd /c must work. Check if you have spaces in the path you are supplying to it.
You can use a parametrized way of passing arguments,
#array = qw("/c", "path/to/xyz.bat");
system("cmd.exe", #array);
The echo directive is not an executable and hence, it errors out.
The same is true of the copy command also. It is not an executable, while xcopy.exe is.
I am having some trouble running ant. Here is a simplified verison of my problem. I have a shell script script1.sh:
export ANT_HOME=/opt/Ant
ant -version
This works. but when I try create another script script2:
cd /location/of/script1
sudo -E ./script1.sh | tee log.txt
I get the error ant: command not found. Does anyone know why this is happening.
Sounds like you're losing your PATH setting after sudo. Try adding echo $PATH in script1.sh to see the before and after values. Or just define script1.sh as
export ANT_HOME=/opt/Ant
${ANT_HOME}/ant -version
Without knowing what shell, or seeing more of the scripts it's hard to tell exactly what is happening. But if you want script2 to know about ANT_HOME you're probably going to need to source or eval script1. See here. Also I know pipes '|' cause Bash to perform operations within sub-shells which can be problematic under certain circumstances (if you're using Bash).
EDIT:
Double check that you are using the version of ant that you think you are:
#!/bin/bash
# Capital A here seems suspicious to me...
export ANT_HOME=/opt/Ant
echo "`${ANT_HOME}/ant -version`"