I am trying to run the mpiexec command from the cell in Jupyter, firstly i am not sure what to write, should i write :
!mpiexec -np 4 mynotebook.ipynb
Secondly, when i run the above mpi command it runs fine but does not give me any output.
Here is what i did :
I installed ipyparallel and started ipcluster from terminal(Mac OS X).
I ran the command:
ipcluster start --profile=mpi -n 4
The servers started succcessfully but it does not show the profile "mpi" in iPython clusters in jupyter homepage.
Have i done all the steps right?
I love Jupyter and i would like to run mpi commands from there. TIA.
Related
I am training myself on Linux with a Raspberry Pi 400 (OS: Raspbian) and I would like to setup my personal config file for apps (notably the Terminal in Zsh). I am able to setup my config onto the terminal with the following:
$ cd .dotfiles
$ ./export.sh
$ cd zsh/
$ source ./zshrc
The problem comes when I quit the terminal or if I open a new terminal or if I reboot the computer, the config is not "saved".
I am not an expert, but for changing bash to zsh and maintaining it as default shell language, I did the following:
$ nano /etc/passwd
and changed the following line from :
pi:x:1000:1000:,,,:/home/pi:/bin/bash
to:
pi:x:1000:1000:,,,:/home/pi:/bin/zsh
This move managed to get me zsh from start every time now. I am wondering if i can directly source my .zshrc with the same technique ... Or would you have any better idea?
I would like to avoid repeating the sourcing steps at every start in the terminal ...
I am exploring docker and I've been using Windows Powershell ISE to run docker commands. I have come across situation where I run certain commands in Powershell but they appear to be stuck, but when I run the same docker command on Command Prompt they work fine. I am using the same machine but different command line tools. Why do certain docker command work in powershell and some don't? The below are just some examples. While most work, some don't appear to be working.
Example 1) docker system prune
In powershell I expected a message like I do in the command prompt below, but it appears to be hanging.
Example 2) docker exec -ti containerName bash
In CMD Prompt, I am able to jump into my running container but not in Powershell
Docker does not support Powershell-ISE:
https://docs.docker.com/docker-for-windows/
Trying to use the KnoxShell example from Windows (using Cmder as shell emulator). When executing the following command, the shell locks, no prompt available and nothing happend. I can only close Shell.
Is that specific to Windows?
λ gradlew.bat -q --no-daemon shell
This is a gradle Application Shell.
You can import your application classes and act on them.
import org.apache.hadoop.gateway.shell.Hadoop;import
org.apache.hadoop.gateway.shell.hdfs.Hdfs;import org.apache.hadoop.gateway.shell.job.Job;import org.apache.hadoop.gateway.shell.workflow.Workflow;import org.apache.hadoop.gateway.shell.yarn.Yarn;import groovy.json.JsonSlurper;import java.util.concurrent.TimeUnit;
===> [import org.apache.hadoop.gateway.shell.Hadoop;import org.apache.hadoop.gateway.shell.hdfs.Hdfs;import org.apache.hadoop.gateway.shell.job.Job;import org.apache.hadoop.gateway.shell.workflow.Workflow;import org.apache.hadoop.gateway.shell.yarn.Yarn;import groovy.json.JsonSlurper;import java.util.concurrent.TimeUnit;]
session = Hadoop.login( https://bi-hadoop-prod-xxxx.bi.services.us-south.bluemix.net:8443/gateway/default, user, password);
slurper = new JsonSlurper();
Groovy Shell (1.8.3, JVM: 1.8.0_51)
Type 'help' or '\h' for help.
-------------------------------------------------------------------------------
groovy:002> hadoop fs -ls /
I have tried the example on a Windows machine and it also hangs for me.
I have updated the README to state that this example does not support Windows and also added some logic to the build script to abort the example if it is being run on a Windows machine.
I am trying to run the following in Jupyter notebook (with Python 2 if it makes a difference):
!head xyz.txt
and I get the following error:
'head' is not recognized as an internal or external command, operable
program or batch file.
Is there anything I need to import to be able to do this?
Might be useful for others.
Use ! followed by terminal command you want to execute. To run a shell command. E.g.,
! pip install some_package
to install the some_package.
An easier way to invoke terminal using jupyter-notebooks is to use magic function %%bash and use the jupyter cell as a terminal:
%%bash
head xyz.txt
pip install keras
git add model.h5.dvc data.dvc metrics.json
git commit -m "Second model, trained with 2000 images"
For Windows it would be %%cmd.
Write it at the beginning of the cell like this :
%%cmd
where python
myprogram "blabla" -x -y -z
You can start the cell with the magic % bash before the rest of your code. There is an example in this blog post, together with a list of some of the most useful magics.
Make sure you run your command in linux shell because there is non such command in windows.
Another option nowadays is the Jupyter kernel for Bash.
I had the same issue. Solved by running
!bash -c "head xyz.txt"
Does anyone know if it is possible to run an IPython/Jupyter notebook non-interactively from the command line and have the resulting .ipynb file saved with the results of the run. If it isn't already possible, how hard would it be to implement with phantomJS, something to turn the kernel on and off, and something to turn the web server on and off?
To be more specific, let's assume I already have a notebook original.ipynb and I want to rerun all cells in that notebook and save the results in a new notebook new.ipynb, but do this with one single command on the command line without requiring interaction either in the browser or to close the kernel or web server, and assuming no kernel or web server is already running.
example command:
$ ipython notebook run original.ipynb --output=new.ipynb
Yes it is possible, and easy, it will (mostly) be in IPython core for 2.0, I would suggest looking at those examples for now.
[edit]
$ jupyter nbconvert --to notebook --execute original.ipynb --output=new.ipynb
It is now in Jupyter NbConvert. NbConvert comes with a bunch of Preprocessors that are disabled by default, two of them (ClearOutputPreprocessor and ExecutePreprocessor) are of interest. You can either enabled them in your (local|global) config file(s) via c.<PreprocessorName>.enabled=True (Uppercase that's python), or on the command line with --ExecutePreprocessor.enabled=True keep the rest of the command as usual.
The --ExecutePreprocessor.enabled=True has convenient --execute alias that can be used on recent version of NbConvert. It can be combine with --inplace if desired
For example, convert to html after running the notebook headless :
$ jupyter nbconvert --to=html --execute RunMe.ipynb
converting to PDF after stripping outputs
$ ipython nbconvert --to=pdf --ClearOutputPreprocessor.enabled=True RunMe.ipynb
This (of course) does work with non-python kernels by spawning a <insert-your-language-here> kernel, if you set --profile=<your fav profile>. The conversion can be really long as it needs to rerun the notebook. You can do notebook to notebook conversion with the --to=notebook option.
There are various other options (timeout, allow errors, ...) that might need to be set/unset depending on use case. See documentation and of course jupyter nbconvert --help, --help-all, or nbconvert online documentation for more information.
Until this functionality becomes part of the core, I put together a little command-line app that does just what you want. It's called runipy and you can install it with pip install runipy. The source and readme are on github.
Run and replace original .ipynb file:
jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --inplace --execute original.ipynb
To cover some features such as parallel workers, input parameters, e-mail sending or S3 input/output... you can install jupyter-runner
pip install jupyter-runner
Readme on github: https://github.com/omar-masmoudi/jupyter-runner
One more way is to use papermill, it has Command Line Interface
Usage example: (you need to specify output path for execution results to be stored)
papermill your_notebook.ipynb logs/yourlog.out.ipynb
You also can specify required params if you wish with -p flag for each param:
papermill your_notebook.ipynb logs/yourlog.out.ipynb -p env "prod" -p tests "e2e"
one more related to papermill reply - https://stackoverflow.com/a/55458141/2957102
You can just run the iPython-Notebook-server via command line:
ipython notebook --pylab inline
This will start the server in non-interactive mode and all output is printed below the code. You can then save the .ipynb-File which includes Code & Output.