I have multiple people working on the same AWS EMR cluster to run some Spark jobs. This is being done through Jupyter Notebooks which are created/modified using the Jupyter extension installed on a SSH Target through vscode. The modules are installed on the base conda environment that is included with the /emr/notebook-env/. Some people can see the correct kernel that is associated with the base conda environment in their vscode window when working on notebooks. However, some don't see this kernel as an option. How do I make sure that everyone's vscode lists the appropriate kernel when they are creating new notebooks or modifying existing notebooks?
Another potential reason this could happen is that the Jupyter exentsion of VSCode is not installed.
To add the Jupyter extension for VSCode, search for it by clicking extensions icon in lefthand toolbar, then searching for Jupyter and installing.
The user having the issue had to update their vscode and that fixed the issue
Related
I'm using jupyter inside a conda environment in vscode on Ubuntu and all works great. However, recently I've experienced issues with AWS access tokens expiring inside my jupyter notebook.
It seems that these access tokens get cached at some level, either in the code environment or somewhere in the jupyter layer, but I couldn't figure out where and how to clean it up.
Example:
I'm using aws-vault to generate SSO credentails, that can be accessed in the environment.
After starting a session, I can access the AWS_SESSION_EXPIRATION from my terminal:
echo $AWS_SESSION_EXPIRATION
2022-04-20T17:40:36Z
when I start vscode, open a terminal, I get the same.
When I activate my environment, and run
python -c "import os;print(os.environ['AWS_SESSION_EXPIRATION'])"
I get the same.
However, when I open a new jupyter notebook and select the same environment, and run
import os
print(os.environ['AWS_SESSION_EXPIRATION'])
I get
2022-04-06T11:09:03Z
I've tried locating any files that could cache the env variables either in the conda env or jupyter extension. I've also tried restarting vscode and the machine itself - all without success.
Can anyone help me clean this up?
Thanks!!
Edit:
Another example:
If I set in my terminal
export FOO=BAR
then I can access $FOO as expected through the terminal and jupy notebook within vscode.
Then I close vscode, run
unset FOO
reboot the machine, start vscode new and open a new notebook with the same conda environment.
Within, running
import os
print(os.environ['FOO'])
STILL returns BAR
There seems to be an issue in the vscode-jupyter extension, that somehow stores the environment variables in the kernel.json file.
At the moment of writing this, the issue is still open. Working in ubuntu, this workaround from the issue's thread works for me:
Close code completely
find ~/.vscode/extensions/ms-toolsai.jupyter*
code .
Run a notebook and it will pick up the changes.
With closed vscode, issue the following command to delete cache:
find ~/.vscode/extensions/ms-toolsai.jupyter* -name kernel.json -delete
Source: https://github.com/microsoft/vscode-jupyter/issues/9774#issuecomment-1110328329
I'm using wsl and it runs codes in vscode pretty fine and I have different libraries which I installed through pip and conda in wsl but when I run that code using vscode itself it doesn't recognize the libraries or even pip itself.
I don't have any other environment.
I should add that I installed the packages globally using conda install ... or pip install ... in base environment and I only have base environment and I run my code through code . and I also have python and remote wsl extensions installed in my vscode.
what can be the problem?
I don't have much personal experience with this, but I found some useful information in this Stack Overflow question (even though it doesn't utilize conda), along with https://code.visualstudio.com/docs/remote/wsl-tutorial#_python-development.
I also found this blog post useful, even if it doesn't cover WSL.
In short, make sure you:
Have installed the Python extension (by Microsoft) in VSCode. This is critical for being able to detect and select the Python interpreter. You don't mention having this in place, so I believe this is your likely problem.
You have done this already, but including it for others who might read this later -- Install the Remote - WSL extension (or the Remote Development extension pack) in VSCode.
You are also doing this already -- Start VSCode from inside your WSL distribution. Alternatively, you can start VSCode from Windows and then select the Remote WSL - Reopen Folder in WSL from the Command Palette (also accessible from the "Remote" Status Bar).
In VSCode, open the Command Palette with Shift+Ctrl+P, search for the Python: Select Interpreter command, and you should find your Conda environment in the list.
After selecting this, you should find that your project is using the interpreter and modules that you have installed via conda.
One thing I did to overcome this issue is go to Extensions -> Local (You should have two tabs there, Local and WSL:DISTRO) DISTRO refers to whatever DISTRO you're using, you will see that some of the local extensions are disabled in the current workspace (WSL) and there is a little cloud icon in the WSL:DISTRO tab that says install Local Extensions in WSL:DISTRO once you click that it will let you choose which extensions to install and you should be good to go!
I reinstalled my vs code, then, when I open the .ipynb in vscode, I also reinstalled the jupyter plugin. But I found the old notebook version opened.
Before I reinstalled these programs, they were still new versions.
I suggest using Anaconda Navigator to download & use a virtual environment that could be easier to use in most cases.To use the virtual environment use -
Ctrl+Shift+P or Python: Select Interpreter command from the Command Palette.However,please make sure to add environment variables to your VS-Code.
Otherwise,best bet may be to use the Jupyter notebook from Anaconda Navigator which pretty much does the same thing
What is the difference between JupyterLab and JupyterHub? How should I decide which one to deploy for users?
JupyterHub is encapsulated environments for multiple users.
JupyterLab is a new version (for the lack of better words) of Jupyter Notebook. It is notebook, text editor and python console together with a file explorer.
Jupyter Notebook plugins may not work with JupyterLab (which is currently beta now).
If you want to use Jupyter Notebook, just use Jupyter Notebook.
If you want multiple users to juse their own Jupyter Notebooks encapsulated in the same machine, Jupyter Hub.
If you want to play with beta software, Jupyter Lab.
I am using vscode with the Python extension and Jupyter extension. For the Python extension, I have several interpreters available, installed by anaconda directly or different virtual environments I have created using conda. Thus, with the Python extension I can run code using different Python versions and packages. However, only one kernel is available for running the jupyter notebooks within vscode:
C:\ProgramData\Anaconda3\python.exe -m ipykernel_launcher
The "jupyter mode" in vscode does not recognize packages I have installed in my virtual environments. Is there a way to start the Python kernel for "jupyter mode" from one of my virtual environments? Or is there another way to solve this problem so that I can use a specific virtual environment in "jupyter mode" within vscode?
Thanks for all help.
The solution was to install the ipykernel for the different conda environments following instructions such as these. These kernels are then automatically identified by vscode.