IPython Notebook: Open/select file with GUI (Qt Dialog) - ipython

When you perform the same analysis in a notebook on different data files, may be handy to graphically select a data file.
In my python scripts I usually implement a QT dialog that returns the file-name of the selected file:
from PySide import QtCore, QtGui
def gui_fname(dir=None):
"""Select a file via a dialog and return the file name.
"""
if dir is None: dir ='./'
fname = QtGui.QFileDialog.getOpenFileName(None, "Select data file...",
dir, filter="All files (*);; SM Files (*.sm)")
return fname[0]
However, running this function from an notebook
full_fname = gui_fname()
causes the kernel to die (and restart):
Interestingly, puttying this 3 command in 3 separate cells works
%matplotlib qt
full_fname = gui_fname()
%matplotlib inline
but when I put those commands in one single cell the kernel dies again.
This prevents to create a function like gui_fname_ipynb() that transparently allows selecting a file with a GUI.
For convenience, I created a notebook illustrating the problem:
Open/select file with GUI (Qt Dialog)
Any suggestion on how to execute a dialog for file selection from within an IPython Notebook?

Using Anaconda 5.0.0 on windows (Python 3.6.2, IPython 6.1.0), the following two options are both working for me.
OPTION 1: Entirely in a Jupyter notebook:
CELL 1:
%gui qt
from PyQt5.QtWidgets import QFileDialog
def gui_fname(dir=None):
"""Select a file via a dialog and return the file name."""
if dir is None: dir ='./'
fname = QFileDialog.getOpenFileName(None, "Select data file...",
dir, filter="All files (*);; SM Files (*.sm)")
return fname[0]
CELL 2:
gui_fname()
This is working for me but it seems a bit...fragile. If I combine these two things into the same cell, it crashes. Or if I omit the %gui qt, it crashes. If I "restart kernel and run all cells", it doesn't work. So I kinda like this other option...
MORE RELIABLE OPTION: Separate script that opens dialog box in a new process
(Based on mkrog code here.)
PUT THE FOLLOWING IN A SEPARATE PYTHON SCRIPT CALLED blah.py:
from sys import executable, argv
from subprocess import check_output
from PyQt5.QtWidgets import QFileDialog, QApplication
def gui_fname(directory='./'):
"""Open a file dialog, starting in the given directory, and return
the chosen filename"""
# run this exact file in a separate process, and grab the result
file = check_output([executable, __file__, directory])
return file.strip()
if __name__ == "__main__":
directory = argv[1]
app = QApplication([directory])
fname = QFileDialog.getOpenFileName(None, "Select a file...",
directory, filter="All files (*)")
print(fname[0])
...AND IN YOUR JUPYTER NOTEBOOK
import blah
blah.gui_fname()

I have a universal code where it does its job without any problem. Here is my sugestion:
try:
from tkinter import Tk
from tkFileDialog import askopenfilenames
except:
from tkinter import Tk
from tkinter import filedialog
Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
filenames = filedialog.askopenfilenames() # show an "Open" dialog box and return the path to the selected file
print (filenames)
hope it can be useful

This behaviour was a bug in IPython:
https://github.com/ipython/ipython/issues/4997
that was fixed here:
https://github.com/ipython/ipython/pull/5077
The function to open a gui dialog should work on current master and on the oncoming 2.0 release.
To date, the last 1.x version (1.2.1) does not include a backport of the fix.
EDIT: The example code still crashes IPython 2.x, see this issue.

Related

How do you embed an ipython console with exec_lines?

I'm trying to embed an ipython console into my command line application.
I have the following:
import IPython
from traitlets.config import Config
c = Config()
c.InteractiveShellApp.exec_lines = [
'import matplotlib.pyplot as plt',
'%matplotlib',
]
return IPython.start_ipython(config=c, user_ns=globals())
However, it seems to completely ignore the "exec_lines" part since plt is not available.
See: Can you specify a command to run after you embed into IPython?
IPython.start_ipython(config=c, user_ns=locals())

How to solve numpy import error when calling Anaconda env from Matlab

I want to execute a Python script from Matlab (on a Windows 7 machine). The libraries necessary are installed in an Anaconda virtual environment. When running the script from command line, it runs flawlessly.
When calling the script from Matlab as follows:
[status, commandOut] = system('C:/Users/user/AppData/Local/Continuum/anaconda3/envs/tf/python.exe test.py');
or with shell commands, I get an Import Error:
commandOut =
'Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
from . import multiarray
ImportError: DLL load failed: The specified path is invalid.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 2, in <module>
import numpy as np
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\__init__.py", line 142, in <module>
from . import add_newdocs
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
from .type_check import *
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified path is invalid.
I already changed the default Matlab Python version to the Anaconda env, but no change:
version: '3.5'
executable: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python.exe'
library: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python35.dll'
home: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf'
isloaded: 1
Just running my test script without importing numpy works. Reloading numpy (py.importlib.import_module('numpy');) didn't work but threw the same error as before.
Does anyone have an idea how to fix this?
So after corresponding with Matlab support I found out that Matlab depends on the path environment (paths which are deliberately not set when using a virtual environment) and therefore numpy fails to find the necessary paths when called from within Matlab (even if the call contains the path to the virtual environment).
The solution is either to call Matlab from within the virtual environment (via command line) or add the missing paths manually in the path environment.
Maybe this information can help someone else.
First Method
You can change the python interpreter with:
pyversion("/home/nibalysc/Programs/anaconda3/bin/python");
And check it with:
pyversion();
You could also do this in a
startup.m
file in your project folder and every time you start MATLAB from this folder the python interpreter will be changed automatically.
Now you can try to use:
py.importlib.import_module('numpy');
Read up the documentation on how to use the integrated python in MATLAB:
Call user defined custom module
Call modified python module
Alternative Method
Alternative method would be to create a
matlab_shell.sh
file with following content, this is basically the appended code from .bashrc when anaconda is installed and asks you if the installer should modify the .bashrc file:
#!/bin/bash
__conda_setup="$(CONDA_REPORT_ERRORS=false '$HOME/path/to/anaconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
\eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
CONDA_CHANGEPS1=false conda activate base
else
\export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda init <<<
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('$HOME/path/to/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "$HOME/path/to/anaconda3/etc/profile.d/conda.sh"
else
export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
conda activate base
eval $2
Then you need to set the MATLAB_SHELL environment variable either before running MATLAB or in MATLAB itself. The best thing in my opinion would be to do it also in the startup.m file like that:
setenv("MATLAB_SHELL", "/path/to/matlab_shell.sh");
Afterwards you can use the system(...) function to run conda python with all your modules installed like that...
String notation:
system("python -c ""python code goes here"");
Char notation:
system('python -c "python code goes here"');
Hope this helps!
Firstly, if you execute your Python script like a regular system command ([status, commandOut] = system('...python.exe test.py'))
the pyversion (and pyenv, since R2019b) got no effect at all. It only matters if you utilize the py. integration, as in the code below (and, in most cases, this is a way better approach).
Currently (I use R2019b update 5) there's a number of pitfalls, that might cause issues similar to yours. I'd recommend to start from the following:
Create a new clean conda environment:
conda create -n test_py36 python=3.6 numpy
Create the following dummy demo1.py:
def dummy_py_method(x):
return x+1
Create the following run_py_code.m:
function run_py_code()
% explicit module import sometimes show more detailed error messages
py.importlib.import_module('numpy');
% to reloads if there would be any changes:
pymodule = py.importlib.import_module('demo1');
py.importlib.reload(pymodule);
% passing data back and forth
x = rand([3 3]);
x_np = py.numpy.array(x);
y_np=pymodule.dummy_py_method(x_np);
y = double(y_np);
disp(y-x);
Create the following before_first_run.m:
setenv('PYTHONUNBUFFERED','1');
setenv('path',['C:\Users\username\Anaconda3\envs\test_py36\Library\bin;'...
getenv('path')]);
pe=pyenv('Version','C:\users\username\Anaconda3\envs\test_py36\pythonw.exe',...
'ExecutionMode','InProcess'...
);
% add "demo1.py" to path
py_file_path = 'W:\tests\Matlab\python_demos\call_pycode\pycode';
if count(py.sys.path,py_file_path) == 0
insert(py.sys.path,int32(0),py_file_path);
end
Run the before_first_run.m first and run the run_py_code.m next.
Notes:
As already mentioned in this answer, one key point is to add the folder, containing the necessary dll files to the %PATH%, before starting python. This could be achieved with setenv from withing Matlab. Usually, the Library\bin is what should be added.
It might be a good idea to try clean officially-supported CPython distribution (e.g. CPython 3.6.8 ). Only install numpy (python -m pip install numpy). To my experience, the setenv is not necessary in this case.
For me, OutOfProcess mode proved to be buggy. Thus, I'd recommend to explicitly setting InProcess mode (for versions before R2019b, the OutOfProcess option is not present, as well as pyenv).
Do not concatenate the two .m files above into one - the py.importlib statements seem to be pre-executed and thus conflict with pyenv.

How to import .py in google Colaboratory?

I want to simplify code. so i make a utils.py , but Google Colaboratory directory is "/content" I read other questions. but this is not my solution
In Google's Colab notebook, How do I call a function from a Python file?
%%writefile example.py
def f():
print 'This is a function defined in a Python source file.'
# Bring the file into the local Python environment.
execfile('example.py')
f()
This is a function defined in a Python source file.
It look likes just using def().
using this, i always write the code in cell.
but i want to this code
import example.py
example.f()
A sample maybe you want:
!wget https://raw.githubusercontent.com/tensorflow/models/master/samples/core/get_started/iris_data.py -P local_modules -nc
import sys
sys.path.append('local_modules')
import iris_data
iris_data.load_data()
I have also had this problem recently.
I addressed the issue by the following steps, though it's not a perfect solution.
src = list(files.upload().values())[0]
open('util.py','wb').write(src)
import util
This code should work with Python 3:
from google.colab import drive
import importlib.util
# Mount your drive. It will be at this path: "/content/gdrive/My Drive/"
drive.mount('/content/gdrive')
# Load your module
spec = importlib.util.spec_from_file_location("YOUR_MODULE_NAME", "/content/gdrive/My Drive/utils.py")
your_module_name = importlib.util.module_from_spec(spec)
spec.loader.exec_module(your_module_name)
import importlib.util
import sys
from google.colab import drive
drive.mount('/content/gdrive')
# To add a directory with your code into a list of directories
# which will be searched for packages
sys.path.append('/content/gdrive/My Drive/Colab Notebooks')
import example.py
This works for me.
Use this if you are out of content folder! hope this help!
import sys
sys.path.insert(0,'/content/my_project')
from example import*
STEP 1. I have just created a folder 'common_module' like shown in the image :
STEP 2 called the required Class from my "colab" code cell,
sys.path.append('/content/common_module/')
from DataPreProcessHelper import DataPreProcessHelper as DPPHelper
My class file 'DataPreProcessHelper.py' looks like this
Add path of 'sample.py' file to system paths as:
import sys
sys.path.append('drive/codes/')
import sample

How to import modules in IPython Clusters

I am trying to import some of my personal modules into my IPython Clusters. I am using Anacondas on Windows Vista 64 bit
from IPython.parallel import Client
rc = Client()
dview = rc[:]
with dview.sync_imports():
import lib.rf
It is giving me this error:
No module named 'lib.rf'
I can import the module in the rest of my IPython notebook, as I have this .bat file to start ipython notebook:
cd C:\Users\Jon\workspace\bf
set PYTHONPATH=%PYTHONPATH%;C:\Users\Jon\workspace\bf
C:\Anaconda\envs\p33\scripts\ipython notebook
I am using this similar code to start my ip clusters:
cd C:\Users\Jon\workspace\bf
set PYTHONPATH=%PYTHONPATH%;C:\Users\Jon\workspace\bf
C:\Anaconda\envs\p33\Scripts\ipcluster start --n=7
Why is this not working?
More info:
If I print out sys.path, I get a list that contains C:\Users\Jon\workspace\bf
If I print out the paths of my clusters, I get the same list:
%px sys.path
['',
'',
'',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\distribute-0.6.28-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\pykalman-0.9.5-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\patsy-0.2.1-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\joblib-0.8.3_r1-py3.3.egg',
'C:\\Users\\Jon\\workspace\\bf',
'C:\\Users\\Jon\\workspace\\bf\\my_numba',
'C:\\Anaconda\\envs\\p33\\python33.zip',
'C:\\Anaconda\\envs\\p33\\DLLs',
'C:\\Anaconda\\envs\\p33\\lib',
'C:\\Anaconda\\envs\\p33',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\Sphinx-1.2.3-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32\\lib',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\Pythonwin',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\runipy-0.1.1-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\setuptools-7.0-py3.3.egg',
'C:\\Anaconda\\envs\\p33\\lib\\site-packages\\IPython\\extensions']
In [45]:
Further analysis:
%px lib.__path__
Out[0:11]: _NamespacePath(['C:\\Anaconda\\envs\\p33\\lib\\site-packages\\win32\\lib'])
lib.__path__
Out[57]: ['.\\lib']
Looks like the ipcluster and notebook are looking at lib in different places. I have tried renaming lib to mylib. It has not helped.
It seems that with dview.sync_imports() is being run someplace other than your IPython Notebook environment and is therefore relying a different PYTHONPATH. It is definitely not being run on one of the cluster engines and so wouldn't expect it to leverage your cluster settings of PYTHONPATH.
I'm thinking you'll need to have that directory in your PYTHONPATH (not your PATH) for the calling python environment because that is the location from which you are importing the modules.
The impact of the bit you have about setting the PYTHONPATH in the DOS shell from which you invoke ipclusters isn't clear to me. I can see that one might expect this to let the engines know about your directory, but I'm wondering if that PYTHONPATH gets initilized to the environment from which you call IPython.parallel.Client.

sharing a namespace across multiple ipython notebooks

I would like to work with several ipython notebooks at once sharing the same namespace. Is there currently (ipython-1.1.0) a way to do this?
I tried creating different notebooks on the same ipython kernel, but the notebooks don't share a namespace. Also, I've been able to use a terminal console alongside a notebook on the same namespace using the answers in Using IPython console along side IPython notebook, but I couldn't find the notebook equivalent of the --existing argument.
Thanks a lot
Unfortunately this no longer works, you get error message ipython.kernel replaced by ipython.parallel.
A less elegant way than above to alter this is to change IPython/frontend/html/notebook/kernelmanager.py around line 273 from
kernel_id = self.kernel_for_notebook(notebook_id)
to
kernel_id = None
for notebook_id in self._notebook_mapping:
kernel_id = self._notebook_mapping[notebook_id]
break
For Anaconda python, replace start_kernel in kernelmanager.py with
def start_kernel(self, kernel_id=None, path=None, **kwargs):
global saved_kernel_id
if saved_kernel_id:
return saved_kernel_id
if kernel_id is None:
kwargs['extra_arguments'] = self.kernel_argv
if path is not None:
kwargs['cwd'] = self.cwd_for_path(path)
kernel_id = super(MappingKernelManager, self).start_kernel(**kwargs)
self.log.info("Kernel started: %s" % kernel_id)
self.log.debug("Kernel args: %r" % kwargs)
self.add_restart_callback(kernel_id,
lambda : self._handle_kernel_died(kernel_id),
'dead',
)
else:
self._check_kernel_id(kernel_id)
self.log.info("Using existing kernel: %s" % kernel_id)
saved_kernel_id = kernel_id
return kernel_id
and add
saved_kernel_id = None
above
class MappingKernelManager(MultiKernelManager):
True IPython gurus, please supply the correct fix. A lot of people using notebooks want the ability to share the kernel, it's natural, because one notebook quickly grows too big to work with a single complex application, so it is easier to be able to break down the application into multiple notebooks.
Also, gurus, while you're listening, it would be nice to have a collapse-expand feature as in Mathematica so you can only view the part of the notebook you care about and you can zoom out the rest.
The IPython Notebook does not have the equivalent of --existing. Notebooks do not share kernels. It is not a limitation of the notebook itself, it is just a design decision made in the notebook server code. The server code can be modified, for instance, to have all notebooks share the same kernel. You can do this with a little monkeypatching in your IPython configuration. Start by creating a profile:
$ ipython profile create singlekernel
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_config.py'
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_qtconsole_config.py'
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_notebook_config.py'
[ProfileCreate] Generating default config file: u'~/.ipython/profile_singlekernel/ipython_nbconvert_config.py'
and edit $(ipython locate profile singlekernel)/ipython_notebook_config.py to contain:
# Configuration file for ipython-notebook.
c = get_config()
import os
import uuid
from IPython.kernel.multikernelmanager import MultiKernelManager
def start_kernel(self, **kwargs):
"""Minimal override of MKM.start_kernel that always returns the same kernel"""
kernel_id = kwargs.pop('kernel_id', str(uuid.uuid4()))
if self.km is None:
self.km = self.kernel_manager_factory(connection_file=os.path.join(
self.connection_dir, "kernel-%s.json" % kernel_id),
parent=self, autorestart=True, log=self.log
)
if not self.km.is_alive():
self.log.info("starting single kernel")
self.km.start_kernel(**kwargs)
else:
self.log.info("reusing existing kernel")
self._kernels[kernel_id] = self.km
return kernel_id
MultiKernelManager.km = None
MultiKernelManager.start_kernel = start_kernel
This just overrides the kernel starting mechanism to start only one kernel and return it at every subsequent request,
rather than starting a new one for each kernel ID.
Now whenever you start the notebook server with
ipython notebook --profile singlekernel
all of the notebooks in that session will share the same kernel.