I'm using the Scala (2.10.3) Process object to run commands. The docs show me how to run a command and then capture the standard output, but I'm running s3cmd and want to see upload progresses. How can I capture the output as if the command were running in a terminal?
Solution:
"s3cmd sync --recursive --delete-removed --progress local/ s3://remote" ! ProcessLogger(line => log.info(line))
Line at a time:
http://www.scala-lang.org/api/2.10.3/#scala.sys.process.ProcessLogger
for vanilla stdout
https://github.com/s3tools/s3cmd/blob/master/S3/Progress.py
Related
I am writing a perl script, and I want to run a simple shell command to use a certain version of NVM:
Here is my code snippet:
print "\n*** Switching to correct nvm environment for dashboard builds\n";
system("nvm use 8.12.0") == 0 or die $?;
But I am getting the following error:
Can't exec "nvm": No such file or directory
Can someone help?
Update (June 30, 2021):
I also tried adding the command:
my $nvm_version = "8.12.0";
system ("bash", "-lic", "nvm use $nvm_version");
But nothing happens:
I'm not familiar with nwm, but I think I get the gist of what it does. And if so, the attempt is fundamentally flawed. Even if you fixed this to run the proper shell so that nvm could run, I believe all the tool does is change the shell's environment variables, a shell you immediately exit. This means it would have no effect even if if it ran successfully.
Again, it this tool does what I think it does, such tool are meant to be used in interactive shells. In other instances, you simply use the path the to correct executable instead of relying on the PATH.
With that in mind, you can use the following to run the command in bash:
# Non-interactive shell.
system("bash", "-c", "nvm use 8.12.0")
or
# Interactive shell.
# This is improper and fragile as interactive shells
# often create aliases that override basic commands.
system("bash", "-ic", "nvm use 8.12.0")
Just to reiterate, at least one of these will allow the command to run (if it normally works from bash), but I believe it's unlikely this will produce the results you expect.
The nvm command is shell function which is different from a shell command. Also the nvm command is not an exported function so it will not be seen by sub shells. For example, in Bash shell:
$ nvm ls
-> v15.0.1
$ my-test-script.sh
./my-test-script.sh: line 3: nvm: command not found
where my-test-script.sh is:
#! /bin/bash
nvm use 16.4
The error nvm: command not found is because nvm is not exported. I can source the script in the current shell context to make it work:
$ source my-test-script.sh
Now using node v16.4.0 (npm v7.18.1)
$ node --version
v16.4.0
So a Perl script cannot change the node version of the current shell, but it can calculate the version and pass it back to shell, which can set the version. For example:
$ nvm use $(perl -E'$v=15.0; print $v')
Now using node v15.0.1 (npm v7.0.3)
I am trying to run the following in Jupyter notebook (with Python 2 if it makes a difference):
!head xyz.txt
and I get the following error:
'head' is not recognized as an internal or external command, operable
program or batch file.
Is there anything I need to import to be able to do this?
Might be useful for others.
Use ! followed by terminal command you want to execute. To run a shell command. E.g.,
! pip install some_package
to install the some_package.
An easier way to invoke terminal using jupyter-notebooks is to use magic function %%bash and use the jupyter cell as a terminal:
%%bash
head xyz.txt
pip install keras
git add model.h5.dvc data.dvc metrics.json
git commit -m "Second model, trained with 2000 images"
For Windows it would be %%cmd.
Write it at the beginning of the cell like this :
%%cmd
where python
myprogram "blabla" -x -y -z
You can start the cell with the magic % bash before the rest of your code. There is an example in this blog post, together with a list of some of the most useful magics.
Make sure you run your command in linux shell because there is non such command in windows.
Another option nowadays is the Jupyter kernel for Bash.
I had the same issue. Solved by running
!bash -c "head xyz.txt"
Here is what I am trying to achieve:
run a matlab command/script that starts a unix terminal and from within that terminal starts external software. Matlab itself should be decoupled from that shell immediately.
On a Unix system, I am currently trying to start an external program from within matlab. I know that I can basically use the matlab command prompt as a terminal by adding an ! in front of every command. However then, the program's output is also displayed within on the matlab command prompt and the program is killed as soon as matlab is closed.
To start an external terminal, call that terminal emulator using the matlab system command. If gnome-terminal is your terminal:
system('gnome-terminal');
To pass parameters to the terminal use -e
system('gnome-terminal -e echo hello World');
This terminal will close immediately after it's finished running. So too keep it open:
system('gnome-terminal -e "bash -c \"echo Hello World; exec bash\""');
Hope this helps. I similar command should work for other terminal emulators beside gnome-terminal.
I'm running a script in solaris 11 with different results depending of the shell used.
The script has an echo redirecting to a file given by an environment value:
echo "STARTING EXEC" >> $FILE
ps. EXEC is just the message the script show, it's not using exec command.
If I execute the script without the variable FILE defined (using /usr/bin/ksh):
./start.sh[10]: : cannot open
and the script continue the execution.
The flags for ksh are:
echo $-
imsuBGEl
But if I change to /usr/xpg4/bin/sh, the script show me the echo in stdout and there is no error shown.
The flags for xpg4 sh are:
echo $-
imsu
I tried to change the flags with set +- (I can't remove El flags, but BG are removed ok), but can't get the same behavior.
Is there anything I can do to get the same result using ksh without cannot open error?
/usr/bin/ksh --version
version sh (AT&T Research) 93u 2011-02-08
I'll want the script keep going, showing the message in stdout, instead of showing the error just like it does now.
Like shellter said in the comments, the good thing to do is to check if the FILE variable is defined before doing anything. This is a script migration from an HPUX to a SOLARIS environment, and client think they must have the same result as before (we unset FILE variable before execution to test it).
You are likely running Solaris 11, not Solaris 64.
Should you want to have your scripts to work under Solaris 11 without having to search everywhere the bogus redirections, you can simply replace all shebangs (first line) by #!/usr/xpg4/bin/sh.
The final solution we are going to take is to install the ksh88 package and use it like default shell (/usr/sunos/bin/ksh). This shell have the same behavior the client had before, and we can let the scripts with no modifications.
The ksh used in solaris 11 is the 93 (http://docs.oracle.com/cd/E23824_01/html/E24456/userenv-1.html#shell-1)
Thanks #jlliagre and #shellter for your help.
Some unix command such as tail -f or starting a python web server (i.e. cherrypy) will produce an endless output, i.e. the only way to stop it is to Ctrl-C. I'm working on a scala application which execute the command like that, my implemetation is:
import scala.sys.process._
def exe(command: String): Unit = {
command !
}
However, as the command produces an endless output stream, the application hangs there until I either terminate it or kill the process started by the command. I also try to add & at the end of the command in order to run it in background but my application still hangs.
Hence, I'm looking for another way to execute a command without hanging my application.
You can use a custom ProcessLogger to deal with output however you wish as soon as it is available.
val proc =
Process(command).run(ProcessLogger(line => (), err => println("Uh-oh: "+err)))
You may kill a process with the destroy method.
proc.destroy
If you are waiting to get a certain output before killing it, you can create a custom ProcessLogger that can call destroy on its own process once it has what it needs.
You may prefer to use lines (in 2.10; the name is changing to lineStream in 2.11) instead of run to gather standard output, since that will give you a stream that will block when no new output is available. Then you wrap the whole thing in a Future, read lines from the stream until you have what you need, and then kill the process--this simplifies blocking/waiting.
Seq("sh", "-c", "tail -f /var/log/syslog > /dev/null &") !
works for me. I think Randall's answer fails because scala is just executing the commands, and can't interpret shell operators like "&". If the command passed to scala is "sh" and the arguments are a complete shell command, we work around this issue. There also seems to be an issue with how scala parses/separates individual arguments, and using a Seq instead of single String works better for that.
The above is equivalent to the unix command:
sh -c 'tail -f /var/log/syslog > /dev/null &'
If you close the descriptor(s) from which you're reading the process' output, it will get a SIGPIPE and (usually) terminate.
If you just don't want the output, redirect to /dev/null:
command arg arg arg >/dev/null 2>&1
Addendum: This pertains only to Unix-alike systems, not Windows.