In IPython, I can load a custom extension using simple command:
%load_ext physics
This will load the file ~/.ipython/extensions/physics.py.
How can I tell IPython to load the extension automatically on startup?
I have added the line to /.ipython/profile_default/startup/import.py, but that does not work:
from numpy import *
%load_ext physics
When I start IPython, I get folowing error:
File "~/.ipython/profile_default/startup/import.py", line 17
%load_ext physics
^
SyntaxError: invalid syntax
From the IPython documentation:
Using extensions
To load an extension while IPython is running, use the %load_ext magic:
In [1]: %load_ext myextension
To load it each time IPython starts, list it in your configuration file:
c.InteractiveShellApp.extensions = [
'myextension'
]
Hope that helps
$ cat ~/.ipython/profile_default/startup/load_extensions.py
get_ipython().run_line_magic('load_ext', 'autoreload')
I confirm that what you attempted originally, but with an .ipy startup file also works:
~/.ipython/profile_default/startup/physics.ipy:
%load_ext physics
Related
I have a Julia module in which I would like to import the Python function sympy.physics.wigner.wigner_9j. My minimal example module is as follows:
module my_module
using PyCall
using SymPy
export test
test()=sympy.physics.wigner.wigner_9j(1,1,1,1,1,1,1,1,1)
end
Then in my Julia notebook running:
using my_module
test()
gives
KeyError: key :physics not found
However adding to the notebook
#pyimport sympy.physics.wigner as sympy_wigner
sympy_wigner.wigner_9j(1,1,1,1,1,1,1,1,1)
gives the correct output. For some reason using #pyimport inside modules gives errors, which I typically avoid by using an __init__ inside my module, e.g adding to my_module.jl
const camb=PyNULL()
function __init__() # this should probably go in SFBBispectrum.jl
copy!(camb, pyimport_conda("camb", "camb", "conda-forge"))
pars=camb.CAMBparams()
end
which allows me to access camb.CAMBparams. Unfortunately I am failing to do something similar for sympy.physics.wigner.wigner_9j.
It has been awhile, but does this help https://github.com/JuliaPy/SymPy.jl/blob/master/src/physics.jl
I want to execute a Python script from Matlab (on a Windows 7 machine). The libraries necessary are installed in an Anaconda virtual environment. When running the script from command line, it runs flawlessly.
When calling the script from Matlab as follows:
[status, commandOut] = system('C:/Users/user/AppData/Local/Continuum/anaconda3/envs/tf/python.exe test.py');
or with shell commands, I get an Import Error:
commandOut =
'Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
from . import multiarray
ImportError: DLL load failed: The specified path is invalid.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 2, in <module>
import numpy as np
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\__init__.py", line 142, in <module>
from . import add_newdocs
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
from .type_check import *
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified path is invalid.
I already changed the default Matlab Python version to the Anaconda env, but no change:
version: '3.5'
executable: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python.exe'
library: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python35.dll'
home: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf'
isloaded: 1
Just running my test script without importing numpy works. Reloading numpy (py.importlib.import_module('numpy');) didn't work but threw the same error as before.
Does anyone have an idea how to fix this?
So after corresponding with Matlab support I found out that Matlab depends on the path environment (paths which are deliberately not set when using a virtual environment) and therefore numpy fails to find the necessary paths when called from within Matlab (even if the call contains the path to the virtual environment).
The solution is either to call Matlab from within the virtual environment (via command line) or add the missing paths manually in the path environment.
Maybe this information can help someone else.
First Method
You can change the python interpreter with:
pyversion("/home/nibalysc/Programs/anaconda3/bin/python");
And check it with:
pyversion();
You could also do this in a
startup.m
file in your project folder and every time you start MATLAB from this folder the python interpreter will be changed automatically.
Now you can try to use:
py.importlib.import_module('numpy');
Read up the documentation on how to use the integrated python in MATLAB:
Call user defined custom module
Call modified python module
Alternative Method
Alternative method would be to create a
matlab_shell.sh
file with following content, this is basically the appended code from .bashrc when anaconda is installed and asks you if the installer should modify the .bashrc file:
#!/bin/bash
__conda_setup="$(CONDA_REPORT_ERRORS=false '$HOME/path/to/anaconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
\eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
CONDA_CHANGEPS1=false conda activate base
else
\export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda init <<<
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('$HOME/path/to/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "$HOME/path/to/anaconda3/etc/profile.d/conda.sh"
else
export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
conda activate base
eval $2
Then you need to set the MATLAB_SHELL environment variable either before running MATLAB or in MATLAB itself. The best thing in my opinion would be to do it also in the startup.m file like that:
setenv("MATLAB_SHELL", "/path/to/matlab_shell.sh");
Afterwards you can use the system(...) function to run conda python with all your modules installed like that...
String notation:
system("python -c ""python code goes here"");
Char notation:
system('python -c "python code goes here"');
Hope this helps!
Firstly, if you execute your Python script like a regular system command ([status, commandOut] = system('...python.exe test.py'))
the pyversion (and pyenv, since R2019b) got no effect at all. It only matters if you utilize the py. integration, as in the code below (and, in most cases, this is a way better approach).
Currently (I use R2019b update 5) there's a number of pitfalls, that might cause issues similar to yours. I'd recommend to start from the following:
Create a new clean conda environment:
conda create -n test_py36 python=3.6 numpy
Create the following dummy demo1.py:
def dummy_py_method(x):
return x+1
Create the following run_py_code.m:
function run_py_code()
% explicit module import sometimes show more detailed error messages
py.importlib.import_module('numpy');
% to reloads if there would be any changes:
pymodule = py.importlib.import_module('demo1');
py.importlib.reload(pymodule);
% passing data back and forth
x = rand([3 3]);
x_np = py.numpy.array(x);
y_np=pymodule.dummy_py_method(x_np);
y = double(y_np);
disp(y-x);
Create the following before_first_run.m:
setenv('PYTHONUNBUFFERED','1');
setenv('path',['C:\Users\username\Anaconda3\envs\test_py36\Library\bin;'...
getenv('path')]);
pe=pyenv('Version','C:\users\username\Anaconda3\envs\test_py36\pythonw.exe',...
'ExecutionMode','InProcess'...
);
% add "demo1.py" to path
py_file_path = 'W:\tests\Matlab\python_demos\call_pycode\pycode';
if count(py.sys.path,py_file_path) == 0
insert(py.sys.path,int32(0),py_file_path);
end
Run the before_first_run.m first and run the run_py_code.m next.
Notes:
As already mentioned in this answer, one key point is to add the folder, containing the necessary dll files to the %PATH%, before starting python. This could be achieved with setenv from withing Matlab. Usually, the Library\bin is what should be added.
It might be a good idea to try clean officially-supported CPython distribution (e.g. CPython 3.6.8 ). Only install numpy (python -m pip install numpy). To my experience, the setenv is not necessary in this case.
For me, OutOfProcess mode proved to be buggy. Thus, I'd recommend to explicitly setting InProcess mode (for versions before R2019b, the OutOfProcess option is not present, as well as pyenv).
Do not concatenate the two .m files above into one - the py.importlib statements seem to be pre-executed and thus conflict with pyenv.
I get the below error when I use pyspark via Zeppelin.
The python & spark interpreters work and all environment variables are set correctly.
print os.environ['PYTHONPATH']
/x01/spark_u/spark/python:/x01/spark_u/spark/python/lib/py4j-0.10.4-src.zip:/x01/spark_u/spark/python:/x01/spark_u/spark/python/lib/py4j-0.10.4-src.zip:/x01/spark_u/spark/python/lib/py4j-0.10.4-src.zip:/x01/spark_u/spark/python/lib/pyspark.zip:/x01/spark_u/spark/python:/x01/spark_u/spark/python/pyspark:/x01/spark_u/zeppelin/interpreter/python/py4j-0.9.2/src:/x01/spark_u/zeppelin/interpreter/lib/python
zepplin-env.sh is set with the below vars
export PYSPARK_PYTHON=/usr/local/bin/python2
export PYTHONPATH=${SPARK_HOME}/python:${SPARK_HOME}/python/lib/py4j-0.10.4-src.zip:${PYTHONPATH}
export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"
See the below log file
INFO [2017-11-01 12:30:42,972] ({pool-2-thread-4}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.PySparkInterpreter
org.apache.zeppelin.interpreter.InterpreterException:
paragraph_1509038605940_-1717438251's Interpreter pyspark not
found
Thank you in advance
I found a workaround for the above issue.The interpreter not found issue does not happen when I create note inside a directory.The issue only happens when I use notes at toplevel.Addionally I foud out that this issue does not happen in 0.7.2 version
Ex :
enter image description here
So, I ran into a weird issue using an ipython notebook and not sure what to do. Normally, when I run a part of the code, if there is an error, I would trace it back, fix it, and then re-run the code. I was doing a similar thing but even after making changes to the code, it looks like nothing is changing!
Here is the example... I am using Python 3.5 so xrange is gone. This then caused an error to be thrown:
XXXX
24 XXXX
25 XXXX
---> 26 for t in xrange(0,len(data),1):
27
28 XXXX
NameError: name 'xrange' is not defined
but after changing my code (which you can see below the difference in line 26), the same error pops up!
XXXX
24 XXXX
25 XXXX
---> 26 for t in range(0,len(data),1):
27
28 XXX
NameError: name 'xrange' is not defined
Any ideas on why this would be happening?
As Thomas K said, you're probably making a change in an external file that was not imported. There is a very useful command in ipython notebook for such cases, called autoreaload. With autoreaload, whenever you modify an external file you do not have to import it again because the extension takes care of it for you. For more information check: ipython autoreload.
Whenever using external files along with Ipython use autoreload. It will reload the external files every time before executing any code in IPython.
Add this at first cell of the IPython.
%load_ext autoreload
%autoreload 2
For me this was due to one of the following:
Cause 1: imported module not updated
Solution:
import importlib
importlib.reload(your_module)
Cause 2: other
Solution: restart the kernel, for jupyter notebook this is how
I have the same problem. I tried jupyter magic autoreload but it didn't work. Finally, I solved it in this way:
in the first cell, add
import My_Functions as my
import importlib
importlib.reload(my)
But notice if the module is imported in this way:
from My_Functions import *
I couldn't reload it properly.
I have the same issue sometimes. I think it has to do with memory - if I have a bunch of dataframes hanging around it seems to cause issues. If I restart the kernel using the Kernel > Restart option the problem goes away.
I have the same problem sometimes. I restarted the kernels but it didn't work.I try to run the cell (ctr+ enter) two or three times. then the result will be displayed according to the updated codes. I hope it helps.
insert new empty cell with + option, go to Kernel, choose Restart & Run All.
Then, fill in the new inserted cell and run again in Kernel, choose Restart & Run All.
It works with me.
I have an GUI-based (TraitsUI/PyQt/Envisage) application written in Python. I would like to spawn an IPython Notebook in which I expose a small API and a number of objects. Those objects include a SQLAlchemy session and a bunch of SQLAlchemy models.
I've looked a lot, but I can't find any examples of this. I can start a notebook:
from IPython.frontend.html.notebook import notebookapp
app = notebookapp.NotebookApp.instance()
app.initialize()
app.start()
and that works well enough (although I'd prefer if 'start' was nonblocking... I assume I can do it in another thread if needed), but I can't alter the namespace.
I've also found examples like this:
from IPython.zmq.ipkernel import IPKernelApp
namespace = dict(z=1010)
kapp = IPKernelApp.instance()
kapp.initialize()
# Update the ns we want with special variables auto-created by the kernel
namespace.update(kapp.shell.user_ns)
# Now set the kernel's ns to be ours
kapp.shell.user_ns = namespace
kapp.start()
But I'm not sure how to actually open the Notebook from here.
Does anybody have any suggestions?
Thanks!
>>> import IPython
>>> z=1010
>>> IPython.embed()
Python 3.5.2 (default, Oct 8 2019, 13:06:37)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: z
Out[1]: 1010