I have a python virtual env generated by poetry somewhere in $HOME/.cache/pypoetry/virtualenvs/pytorch-exploration-7_nwJ185-py3.8. In this venv, I have torch installed as a library. import statements involving torch work while running my program with poetry run. To properly run the program, poetry needs
#!/usr/bin/env python3.8
Despite that I set VScode to use this interpreter $HOME/.cache/pypoetry/virtualenvs/pytorch-exploration-7_nwJ185-py3.8/bin/python3.8, I am still getting "unresolved import 'torch'" from VScode and I am unable to easily look at torch's objects definitions.
I read several hacks like this one but I'd rather find a way to load the actual libraries of the venv to avoid any version mismatch or side effects. Does VScode have the ability to load libraries from the venv based on the declared interpreter?
Related
I am currently working on project,where I use pytorch as deep learning environment. I have serious problem, that .... "When I use anaconda jupyter notebook terminal to run my *.py files. Then, it run properly. But, when I use the same environment within vs-code via python_interpretor, I can not get result and give me error "torch module can not found".
Can anyone have solution?. Actually, I see the packages using "conda list" and it includes torchvision inside environment. I also tried to install torch in base environment But, it does not work.
Please give me some solution. Thank you for your consideration.
I had started with udacity deep learning course and was setting up environments. I think the kernel notebook uses does not use python from conda environment. Following are some of the results of things I have tried.
Started conda environment
source activate tensorflow
With python terminal inside conda environment from linux terminal:
import sys
sys.executable
>>> '/home/username/anaconda2/envs/tensorflow/bin/python'
Also tensorflow gets imported with python shell
With ipython terminal inside conda environment, it shows same executable path. and tensorflow gets imported inside ipython shell.
However with jupyter notebook when I execute a cell in notebook, tensorflow module cannot be found. Also terminal spawned from notebook shows executable path of global python installation which is in anaconda/bin directoty, not of environment I had created from which I started the notebook
'/home/username/anaconda2/bin/python'
However conda environment of shell is still tensorflow
conda info --envs
# conda environments:
#
tensorflow * /home/username/anaconda2/envs/tensorflow
root /home/username/anaconda2
Does that mean kernel is linked to python installation in this location and not in conda env? How to link the same?
There is some more nuance to this question that is good to clarify. Each notebook is bound to a particular kernel. With the latest 4.0 release of Anaconda we (Continuum) have bundled a Conda-environment-aware extension that will try to associate a Notebook with a particular Conda environment. If that cannot be found then the "default" environment (or "root" environment) will be used. In your case you have a Notebook that is, I am guessing, asking for the default (or "root") environment, and so Jupyter starts a kernel in that environment, and not in the environment from which the Jupyter server was started. You can change the associated kernel by going to the Kernel->Change kernel menu and picking your tensorflow environment's kernel, along the lines of this:
Or when you create a new Notebook you can pick at that time which Conda environment's kernel should back the Notebook (note that one Conda environment can have multiple kernels available, e.g. Python and R):
We appreciate that this can be a common cause of confusion, especially when sharing notebooks, since the person who shared it either used the "default" kernel (probably called just "Python"), or they were using a Conda environment with a different name. We are working on ways to make this smoother and less confusing, but if you have suggestions for expected/desired behavior, please let us know (GitHub issue to https://github.com/ContinuumIO/anaconda-issues/issues/new is the best way to do this)
I'm trying to import a self-defined module in a Jupyter notebook using PyCharm (2016.1). However, I always get "ImportError: No module named xxx". Importing packages like NumPy or Matplotlib works fine. The self-defined module and the notebook are in the same directory and I've tried to set the directory as sources root. How can I fix this? Thanks a lot!
If you run the following in your notebook...
import sys
sys.path
...and you don't see the path to the directory containing the packages/modules, there are a couple ways around it. I can't speculate why this might happen in this example. I have seen some discrepancies in the results of sys.path when running Jupyter locally from PyCharm on OS X vs. on a managed Linux service.
An easy if hacky workaround is to set the sys path in your notebook to reflect where the packages/modules are rooted. For example, if your notebook was in a subdirectory from where the packages or modules are and sys.path only reflects that subdirectory:
import sys
sys.path.append("../")
The point is that sys.path must include the the directory the packages and modules are rooted in so the path you append will depend on the circumstances.
Perhaps a more proper solution, if you are using a virtualenv as your project interpreter, is to create a setup.py for your project and install the project as an editable package with pip. E.g. pip install -e . Then as long as Jupyter is running from that virtualenv there shouldn't be any issues with imports.
One ugly gotcha I ran into on OS X was Jupyter referencing the wrong virtualenv when started. This should also be apparent by inspecting the results of sys.path. I don't really know how I unintentionally managed set this but presume it was due to futzing around my first time getting Jupyter working in PyCharm. Instead of starting Jupyter with the local virtual env it would run with the one defined in ~/Library/Jupyter/kernels/.python/kernel.json. I was able to clear it by cleaning out that directory, e.g. rm -r ~/Library/Jupyter/kernels/.python.
As stated by Thomas in the comments make sure that your notebook serving path and project path are same. When you start your notebook in pycharm you should get something like this :
Serving notebooks from local directory: <path to your project root folder>
Background
I use Anaconda's IPython on my mac and it's a great tool for data exploration and debugging. However, when I wish to use IPython for my programs that require virtualenv (e.g. a Django web app), I don't want to have to reinstall IPython every time.
Question
Is there a way to use my local IPython while also using the rest of my virtualenv packages? (i.e. just make IPython the exception to virtualenv packages so that the local IPython setup is available no matter what) If so, how would you do this on a mac? My guess is that it would be some nifty .bash_profile changes, but my limited knowledge with it hasn't been fruitful. Thanks.
Example Usage
Right now if I'm debugging a program, I'd use the following:
import pdb
pdb.set_trace() # insert this to pause program and explore at command line
This would bring it to the command line (that I wish was IPython)
If you have a module in your local Python and not in the virtualenv, it will still be available in the virtualenv. Unless you shadow it with another virtualenv version. Did you try to launch your local IPython from a running virtualenv that didn't have an IPython? It should work.
Will, I assume you are using Anaconda's "conda" package manager? (Which combines the features of pip and virtualenv). If so you should be aware that many parts of it does not work completely like the tools it is replacing. E.g. if you are using conda create -n myenv to create your virtual environment, this is different from the "normal" virtualenv in a number of ways. In particular, there is no "global/default" packages: Even the default installation is essentially an environment ("root") like all other environments.
To obtain the usual virtualenv behavior, you can create your environments by cloning the root environment: conda create -n myenv --clone root. However, unlike for regular virtualenv, if you make changes to the default installation (the "root" environment in conda) these changes are not reflected in the environments that were created by cloning the root environment.
An alternative to cloning the root is to keep an updated list of "default packages" that you want to be available in new environments. This is managed by the create_default_packages option in the condarc file.
In summary: Don't treat your conda environments like regular python virtualenvs - even though they appear deceptively similar in many regards. Hopefully at some point the two implementations will converge.
All my install for python were working pretty well since soon. Now, I have a weird issue that generates import errors for some of my Python scripts and when using Ipython those import performs as expected.
This issue does not depend on the working dir (tested this several times with the same result). And I didn't change anything in my .bashrc regarding PYTHONPATHs.
So this might be a noob question, but is there an option in Ipython to display everything Ipython is doing when I run a command, so that I can track which is the package Ipython succesfully imported?
The packages I have trouble with are actually yaml, matplotlib cloud Image. Is it possible that an update from 2.6 to 2.7 is responsible for that? (I work on OSX Lion and most of the packages are installed via macports and updated using the upgrade outdated command).
As I think I can probably fix this myself with the command I asked for, don't bother seeking further for package compatibility or anything if you got the Ipython command that might suits me.
The difference of imports between python and ipython resulted from a cross defined python reference library (when installing most of the python libraries with macports and some without any known ports with other package managers).
Reinstalling the whole python libraries using the process described in http://astrofrog.github.io/macports-python/
corrects the issue and it is preferable to install manually the libraries that don't have any known ports (for instance with a setup build/install process).