I am loading a remote kernel in a local Jupyter notebook with the rk python package. It works fine except no environment variables of the remote are loaded. The question is then:
How to load the remote environment variables when using a remote kernel? Is there something in the asociated kernel.json to modify?
Related
I'm using jupyter inside a conda environment in vscode on Ubuntu and all works great. However, recently I've experienced issues with AWS access tokens expiring inside my jupyter notebook.
It seems that these access tokens get cached at some level, either in the code environment or somewhere in the jupyter layer, but I couldn't figure out where and how to clean it up.
Example:
I'm using aws-vault to generate SSO credentails, that can be accessed in the environment.
After starting a session, I can access the AWS_SESSION_EXPIRATION from my terminal:
echo $AWS_SESSION_EXPIRATION
2022-04-20T17:40:36Z
when I start vscode, open a terminal, I get the same.
When I activate my environment, and run
python -c "import os;print(os.environ['AWS_SESSION_EXPIRATION'])"
I get the same.
However, when I open a new jupyter notebook and select the same environment, and run
import os
print(os.environ['AWS_SESSION_EXPIRATION'])
I get
2022-04-06T11:09:03Z
I've tried locating any files that could cache the env variables either in the conda env or jupyter extension. I've also tried restarting vscode and the machine itself - all without success.
Can anyone help me clean this up?
Thanks!!
Edit:
Another example:
If I set in my terminal
export FOO=BAR
then I can access $FOO as expected through the terminal and jupy notebook within vscode.
Then I close vscode, run
unset FOO
reboot the machine, start vscode new and open a new notebook with the same conda environment.
Within, running
import os
print(os.environ['FOO'])
STILL returns BAR
There seems to be an issue in the vscode-jupyter extension, that somehow stores the environment variables in the kernel.json file.
At the moment of writing this, the issue is still open. Working in ubuntu, this workaround from the issue's thread works for me:
Close code completely
find ~/.vscode/extensions/ms-toolsai.jupyter*
code .
Run a notebook and it will pick up the changes.
With closed vscode, issue the following command to delete cache:
find ~/.vscode/extensions/ms-toolsai.jupyter* -name kernel.json -delete
Source: https://github.com/microsoft/vscode-jupyter/issues/9774#issuecomment-1110328329
I am trying to run some notebooks in my virtual environment in the VSCode (remotely connected). I install the venv as usual via python3 -m venv <venv-name>, activate it and install all the needed modules. When I run which ipython I get the one from the venv so I install the kernel via ipython kernel install --name "<name>" --user and it is successfully created in ~/.local/share/jupyter/kernels/ directory and the kernel.json points to the venv python. Then I open the VSCode and select both the Python: Select Interpreter and Jupyter: Select Interpreter to start Jupyter server to point to the virtual environment's python, sth. like .../<venv-name>/bin/python3.
However, when I try to run the cell it wants me to select kernel (I can also do it myself in the upper right corner of the VSCode), but my newly created kernel is not there. There are only two (same) ones from usr/bin/python.
It is really strange since twice in two days my kernel magically appeared for one notebook and worked as desired, but when I opened a new notebook, my kernel was gone again. I tried to remove/reinstall kernels, venvs, VSCode's Python and Jupyter extensions but nothing helped. Any suggestions?
For now, I start the kernel in remote command-line via jupyter notebook --no-browser --ip=<ip> and then insert the connection link to Jupyter Server in the bottom right corner of the VSCode status bar but am wondering if there is an easier way since all the stuff (except VSCode) is on a remote machine?
This way is not easy. You can set up Jupyter Kernel easily.
Firstly, using ssh to connect to the remote server.
Secondly, open Command Palette (⇧⌘P) and enter Python: Select Interpreter, you can directly connecting to remote kernel.
resource: https://code.visualstudio.com/docs/datascience/jupyter-notebooks
I have VSCode set up with the "Remote - SSH" extension, and I am able to run a local VSCode window as the face of a full remote environment. Really cool.
However, I would also like to be able to set up a local VSCode environment which loads files from my local filesystem but runs them on a remote Python interpreter (via an SSH connection).
I am hoping I can run all the regular interactive Python REPL stuff using the remote kernel (e.g. show variables, etc). It seems like I should be able to somehow add a remote kernel and then select it via "Python: Select Interpreter", but I can't figure out how.
In the past I was able to configure such a setup in JupyterLab using the "remote_ikernel" package.
Is this possible in VSCode? I've found tons of documentation on the full remote setup, but haven't located information on this hybrid kind of set up.
I found a related bug report which discusses mixing local and remote filesystems (not interpreters).
I suspect that the same difficulties it describes for filesystems may be similar reasons for why local files can not be easily fed into remote interpreters.
The full thread is here for reference:
https://github.com/microsoft/vscode-remote-release/issues/706
The specific quote:
Having a mixed workspace would require that a workspace extension can be active both locally and remotely. Currently an extension can only be active once, either locally or remotely. So this is currently not possible and we have no plans to support this.
The recommendation is to have separate windows open for the local and remote workspace.
So it sound like this is a feature that is not currently supported.
I had started with udacity deep learning course and was setting up environments. I think the kernel notebook uses does not use python from conda environment. Following are some of the results of things I have tried.
Started conda environment
source activate tensorflow
With python terminal inside conda environment from linux terminal:
import sys
sys.executable
>>> '/home/username/anaconda2/envs/tensorflow/bin/python'
Also tensorflow gets imported with python shell
With ipython terminal inside conda environment, it shows same executable path. and tensorflow gets imported inside ipython shell.
However with jupyter notebook when I execute a cell in notebook, tensorflow module cannot be found. Also terminal spawned from notebook shows executable path of global python installation which is in anaconda/bin directoty, not of environment I had created from which I started the notebook
'/home/username/anaconda2/bin/python'
However conda environment of shell is still tensorflow
conda info --envs
# conda environments:
#
tensorflow * /home/username/anaconda2/envs/tensorflow
root /home/username/anaconda2
Does that mean kernel is linked to python installation in this location and not in conda env? How to link the same?
There is some more nuance to this question that is good to clarify. Each notebook is bound to a particular kernel. With the latest 4.0 release of Anaconda we (Continuum) have bundled a Conda-environment-aware extension that will try to associate a Notebook with a particular Conda environment. If that cannot be found then the "default" environment (or "root" environment) will be used. In your case you have a Notebook that is, I am guessing, asking for the default (or "root") environment, and so Jupyter starts a kernel in that environment, and not in the environment from which the Jupyter server was started. You can change the associated kernel by going to the Kernel->Change kernel menu and picking your tensorflow environment's kernel, along the lines of this:
Or when you create a new Notebook you can pick at that time which Conda environment's kernel should back the Notebook (note that one Conda environment can have multiple kernels available, e.g. Python and R):
We appreciate that this can be a common cause of confusion, especially when sharing notebooks, since the person who shared it either used the "default" kernel (probably called just "Python"), or they were using a Conda environment with a different name. We are working on ways to make this smoother and less confusing, but if you have suggestions for expected/desired behavior, please let us know (GitHub issue to https://github.com/ContinuumIO/anaconda-issues/issues/new is the best way to do this)
I have installed Anaconda for my Machine Learning course. I'm using it as IPython (Jupyter) notebook, in which we have lessons. OS is Ubuntu 14.04 LTS. Basically, I always run it from Terminal with:
jupyter notebook
I have created new environment called su_env from root environment (exact copy) with one package added. Now, I'm wondering: how can I set environment su_env as default one? I have dozen of notebooks so it's annoying to set up each time for every notebook the environment, in "web" GUI of Jupyter.
EDIT: I'm interested in a solution where you don't have to set environment before running notebook. My logic is that, somehow, automagically, jupyter sets root environment on its own while starting up. Because of that, I'm wondering is it possible to set some config file or something so jupyter sets su_env instead of root. Also, if you know that's not possible (and why), I would like to know that.
First activate the conda environment from the command line, then launch the notebook server.
For example:
$ source activate env_name
$ jupyter notebook
Note: This might only work with environments that were created from within Jupyter Notebook, not environments that were created using conda create on the command line.
In your ~/.bashrc, include the line:
alias jupyter="source activate su_env; jupyter"
This will condense the two commands into one, and you will activate su env whenever you call jupyter notebook or lab or whatever
Edit your bashrc and add source activate su_env then that env will always be active. To switch back to to root (or any other env) source activate env_name
You can use this on conda prompt:
conda activate env_name
jupyter notebook
source activate env_name gives me an error: 'source' is not recognized as an internal or external command, operable program, or batch file.