How Can I use my GPU on Ipython Notebook? - ipython

OS : Ubuntu 14.04LTS
Language : Python Anaconda 2.7 (keras, theano)
GPU : GTX980Ti
CUDA : CUDA 7.5
I wanna run keras python code on IPython Notebook by using my GPU(GTX980Ti)
But I can't find it.
I want to test below code. When I run it on to Ubuntu terminal,
I command as below (It uses GPU well. It doesn't have any problem)
First I set the path like below
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
Second I run the code as below
THEANO_FLAGS='floatX=float32,device=gpu0,nvcc.fastmath=True' python myscript.py
And it runs well.
But when i run the code on pycharm(python IDE) or
When I run it on Ipython Notebook, It doesn't use gpu.
It only uses CPU
myscript.py code is as below.
from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time
vlen = 10 * 30 * 768 # 10 x #cores x # threads per core
iters = 1000
rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in xrange(iters):
r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
print('Used the cpu')
else:
print('Used the gpu')
To solve it, I force the code use gpu as below
(Insert two lines more on myscript.py)
import theano.sandbox.cuda
theano.sandbox.cuda.use("gpu0")
Then It generate the error like below
ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.
how to do it??? I spent two days..
And I surely did the way of using '.theanorc' file at home directory.

I'm using theano on an ipython notebook making use of my system's GPU. This configuration seems to work fine on my system.(Macbook Pro with GTX 750M)
My ~/.theanorc file :
[global]
cnmem = True
floatX = float32
device = gpu0
Various environment variables (I use a virtual environment(macvnev):
echo $LD_LIBRARY_PATH
/opt/local/lib:
echo $PATH
/Developer/NVIDIA/CUDA-7.5/bin:/opt/local/bin:/opt/local/sbin:/Developer/NVIDIA/CUDA-7.0/bin:/Users/Ramana/projects/macvnev/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
echo $DYLD_LIBRARY_PATH
/Developer/NVIDIA/CUDA-7.5/lib:/Developer/NVIDIA/CUDA-7.0/lib:
How I run ipython notebook (For me, the device is gpu0) :
$THEANO_FLAGS=mode=FAST_RUN,device=gpu0,floatX=float32 ipython notebook
Output of $nvcc -V :
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Thu_Sep_24_00:26:39_CDT_2015
Cuda compilation tools, release 7.5, V7.5.19
From your post, probably you've set the $PATH variable wrong.

Related

How to use Micropython Classes in separate files

Getting started with MicroPython and having problems with classes in separate files:
In main.py:
import clientBase
import time
if __name__ == "__main__":
time.sleep(15) # Delay to open Putty
print("Starting")
print("Going to class")
cb = clientBase.ClientBaseClass
cb.process()
In clientBase.py:
class ClientBaseClass:
def __init__(self):
print("init")
def process(self):
print("Process")
Compiles and copies to Pico without errors but does not run. Putty output: No idea how to run Putty (or other port monitor) without blocking port!
MPY: soft reboot
Traceback (most recent call last):
Thanks
Python Conslole:
"C:\Users\jluca\OneDrive\Apps\Analytical Engine\Python\Client\venv\Scripts\python.exe" "C:\Program Files\JetBrains\PyCharm Community Edition 2021.2.4\plugins\python-ce\helpers\pydev\pydevconsole.py" --mode=client --port=59708
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['C:\Users\jluca\OneDrive\Apps\Analytical Engine\Python\Client', 'C:\Users\jluca\AppData\Roaming\JetBrains\PyCharmCE2021.2\plugins\intellij-micropython\typehints\stdlib', 'C:\Users\jluca\AppData\Roaming\JetBrains\PyCharmCE2021.2\plugins\intellij-micropython\typehints\micropython', 'C:\Users\jluca\AppData\Roaming\JetBrains\PyCharmCE2021.2\plugins\intellij-micropython\typehints\rpi_pico', 'C:/Users/jluca/OneDrive/Apps/Analytical Engine/Python/Client'])
PyDev console: starting.
Python 3.10.3 (tags/v3.10.3:a342a49, Mar 16 2022, 13:07:40) [MSC v.1929 64 bit (AMD64)] on win32
The first problem I see here is that you're not properly instantiating the ClientBaseClass object. You're missing parentheses here:
if __name__ == "__main__":
time.sleep(15) # Delay to open Putty
print("Starting")
print("Going to class")
cb = clientBase.ClientBaseClass # <-- THIS IS INCORRECT
cb.process()
This is setting the variable cb the class ClientBaseClass, rather than creating a new object of that class.
You need:
if __name__ == "__main__":
time.sleep(15) # Delay to open Putty
print("Starting")
print("Going to class")
cb = clientBase.ClientBaseClass()
cb.process()
I don't know if that's your only problem or not; seeing your traceback will shed more details on the problem.
If I fix that one problem, it all seems to work. I'm using ampy to transfer files to my Pico board (I've also repeated the same process using the Thonny edit, which provides a menu-driven interface for working with Micropython boards):
$ ampy -p /dev/usbserial/3/1.4.2 put main.py
$ ampy -p /dev/usbserial/3/1.4.2 put clientBase.py
$ picocom -b 115200 /dev/usbserial/3/1.4.2
I press return to get the Micropython REPL prompt:
<CR>
>>>
And then type CTRL-D to reset the board:
>>> <CTRL-D>
MPY: soft reboot
And then the board comes up, the code executes as expected:
<pause for 15 seconds>
Starting
Going to class
init
Process
MicroPython v1.18 on 2022-01-17; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>>
(note that if you replace MicroPython with CircuitPython,the Pico will show up as a drive and you can just drag-and-drop files on it.)
Tried micropython and circuitpython with Pycharm, Thonny and VisualStudio code. The only thing that reliably works is CircuitPython with Mu editor. I think its all about the way the .py files are copied to the Pico board and life's too short to do more diagnostics. Mu is pretty basic but it works! Thanks for the help.

VcXsrv WSL X server closes unexpectedly. Do I need to disable GPU?

I am trying to run some code with pybullet. I am on windows 10, have the latest vscode, and I am using WSL remote on vscode with ubuntu 18.04 LTS. I have a GTX 2070 graphics card. I just want to see this work, I've been trying to fix it for the last 3 hours.
First, here is the code I am trying to run in WSL:
import numpy as np
import pybullet as pb
physicsClient = pb.connect (pb.GUI)
#load plane
import pybullet_data
pb.setAdditionalSearchPath(pybullet_data.getDataPath())
planeId = pb.loadURDF('plane.urdf')
#load visual shape
visualShapeId = pb.createVisualShape(
shapeType=pb.GEOM_MESH,
fileName='random_urdfs/000/000.obj',
rgbaColor=None,
meshScale=[0.1, 0.1, 0.1])
collisionShapeId = pb.createCollisionShape(
shapeType=pb.GEOM_MESH,
fileName='random_urdfs/000/000_coll.obj',
meshScale=[0.1, 0.1, 0.1])
multiBodyId = pb.createMultiBody(
baseMass=1.0,
baseCollisionShapeIndex=collisionShapeId,
baseVisualShapeIndex=visualShapeId,
basePosition=[0, 0, 1],
baseOrientation=pb.getQuaternionFromEuler([0, 0, 0]))
I get no errors, but the X server window will pop up (black) and close immediately. I read that you need to disable your GPU with WSL, but I am scared of messing up my PC. I would only want to disable it for when I need to see graphics / use the X server, not for all WSL applications.
Here is what shows in my bash script:
user#DESKTOP-######:~/program$ python3 openAI.py
pybullet build time: Sep 22 2020 00:54:31
startThreads creating 1 threads.
starting thread 0
started thread 0
argc=2
argv[0] = --unused
argv[1] = --start_demo_name=Physics Server
ExampleBrowserThreadFunc started
X11 functions dynamically loaded using dlopen/dlsym OK!
X11 functions dynamically loaded using dlopen/dlsym OK!
Creating context
Failed to create GL 3.3 context ... using old-style GLX context
Indirect GLX rendering context obtained
Making context current
GL_VENDOR=NVIDIA Corporation
GL_RENDERER=GeForce RTX 2070 SUPER/PCIe/SSE2
GL_VERSION=1.4 (4.6.0 NVIDIA 451.67)
GL_SHADING_LANGUAGE_VERSION=(null)
pthread_getconcurrency()=0
Version = 1.4 (4.6.0 NVIDIA 451.67)
Vendor = NVIDIA Corporation
Renderer = GeForce RTX 2070 SUPER/PCIe/SSE2
Segmentation fault (core dumped)
user#DESKTOP-######:~/program$
#Emilio, I have got this working without any changes to the GPU, using the following process:
I used the VcXsrv application set up in the same way as this tutorial : https://jack-kawell.com/2020/06/12/ros-wsl2/ where crucially Native openGL is unchecked.
Export your ip address as in the tutorial, however instead of 'export DISPLAY={your_ip_address}:0.0', go to the VcXsrv window (which should be blank at this point) and replace :0.0 with whatever number of display is given. So for Display DESKTOP-1234AB:1.0 you would enter 'export DISPLAY={your_ip_address}:1.0'
In the linux terminal enter: export LIBGL_ALWAYS_INDIRECT=0
You can check that this has made an effect by entering: glxinfo
Which should print out:
direct rendering: yes
When you run your python program it should open up in the VcXsrv window. For me there was no cursor visible but I could still interact with the object as if I did have a cursor.

MATLAB — Unable to Import cv2 Library

I'm a beginner at OpenCV, and trying to run an open-source program.
http://asrl.utias.utoronto.ca/code/gpusurf/index.html
I currently have the Computer Vision Toolbox OpenCV Interface 20.1.0 installed and Computer Vision Toolbox 9.2.
I cannot run this simple open-source feature matching algorithm without encountering errors.
import cv2
import matplotlib.pyplot as plt
%matplotlib inline
% read images
img1 = cv2.imread('[INSERT PATH #1]');
img2 = cv2.imread('[INSERT PATH #2]');
img1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY);
img2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY);
%sift
sift = cv2.xfeatures2d.SIFT_create();
keypoints_1, descriptors_1 = sift.detectAndCompute(img1,None);
keypoints_2, descriptors_2 = sift.detectAndCompute(img2,None);
len(keypoints_1), len(keypoints_2)
The following message is returned:
Error: File: Keypoints.m Line: 1 Column: 8
The import statement 'import cv2' cannot be found or cannot be imported. Imported names must end with '.*' or be
fully qualified.
However, when I remove Line 1, I instead get the following error.
Error: File: Keypoints.m Line: 2 Column: 8
The import statement 'import matplotlib.pyplot' cannot be found or cannot be imported. Imported names must end
with '.*' or be fully qualified.
Finally, following the error message only results in a sequence of further errors from the cv2 library. Any ideas?
That's because the code you've used isn't MATLAB code, it's python code.
As per the website you've linked:
From within Matlab
The parallel implementation coded in Matlab can be run by using the surf_find_keypoints() function. The output keypoints can be sorted by strength using surf_best_n_keypoints(), and plotted using surf_plot_keypoints().
Check that you've downloaded the correct files and try again.
Furthermore, the Matlab OpenCV Interface is designed to integrate C++ OpenCV code, not python. Documentations here.
Yes, it is correct that this is Python code. I would recommend checking your dependencies/libraries. The PyCharm IDE is what I personally use since it takes care of all the libraries easily.
If you do end up trying out PyCharm click on the red icon when hovering on CV2. It’ll then give you a prompt to download the library.
Edit:
Using Python some setup can be done. Using pip:
Install opencv-python
pip install opencv-python
Install opencv-contrib-python
pip install opencv-contrib-python
Unfortunately, there is some issue with the sift feature since by default it is excluded from newer free versions of OpenCV.
sift = cv2.xfeatures2d.SIFT_create() not working even though have contrib installed
import cv2
Image_1 = cv2.imread("Image_1.png", cv2.IMREAD_COLOR)
Image_2 = cv2.imread("Image_2.jpg", cv2.IMREAD_COLOR)
Image_1 = cv2.cvtColor(Image_1, cv2.COLOR_BGR2GRAY)
Image_2 = cv2.cvtColor(Image_2, cv2.COLOR_BGR2GRAY)
sift = cv2.SIFT_create()
keypoints_1, descriptors_1 = sift.detectAndCompute(Image_1,None)
keypoints_2, descriptors_2 = sift.detectAndCompute(Image_2,None)
len(keypoints_1), len(keypoints_2)
The error I received:
"/Users/michael/Documents/PYTHON/Test Folder/venv/bin/python" "/Users/michael/Documents/PYTHON/Test Folder/Testing.py"
Traceback (most recent call last):
File "/Users/michael/Documents/PYTHON/Test Folder/Testing.py", line 9, in <module>
sift = cv2.SIFT_create()
AttributeError: module 'cv2.cv2' has no attribute 'SIFT_create'
Process finished with exit code 1

How to solve numpy import error when calling Anaconda env from Matlab

I want to execute a Python script from Matlab (on a Windows 7 machine). The libraries necessary are installed in an Anaconda virtual environment. When running the script from command line, it runs flawlessly.
When calling the script from Matlab as follows:
[status, commandOut] = system('C:/Users/user/AppData/Local/Continuum/anaconda3/envs/tf/python.exe test.py');
or with shell commands, I get an Import Error:
commandOut =
'Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
from . import multiarray
ImportError: DLL load failed: The specified path is invalid.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 2, in <module>
import numpy as np
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\__init__.py", line 142, in <module>
from . import add_newdocs
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
from .type_check import *
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified path is invalid.
I already changed the default Matlab Python version to the Anaconda env, but no change:
version: '3.5'
executable: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python.exe'
library: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python35.dll'
home: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf'
isloaded: 1
Just running my test script without importing numpy works. Reloading numpy (py.importlib.import_module('numpy');) didn't work but threw the same error as before.
Does anyone have an idea how to fix this?
So after corresponding with Matlab support I found out that Matlab depends on the path environment (paths which are deliberately not set when using a virtual environment) and therefore numpy fails to find the necessary paths when called from within Matlab (even if the call contains the path to the virtual environment).
The solution is either to call Matlab from within the virtual environment (via command line) or add the missing paths manually in the path environment.
Maybe this information can help someone else.
First Method
You can change the python interpreter with:
pyversion("/home/nibalysc/Programs/anaconda3/bin/python");
And check it with:
pyversion();
You could also do this in a
startup.m
file in your project folder and every time you start MATLAB from this folder the python interpreter will be changed automatically.
Now you can try to use:
py.importlib.import_module('numpy');
Read up the documentation on how to use the integrated python in MATLAB:
Call user defined custom module
Call modified python module
Alternative Method
Alternative method would be to create a
matlab_shell.sh
file with following content, this is basically the appended code from .bashrc when anaconda is installed and asks you if the installer should modify the .bashrc file:
#!/bin/bash
__conda_setup="$(CONDA_REPORT_ERRORS=false '$HOME/path/to/anaconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
\eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
CONDA_CHANGEPS1=false conda activate base
else
\export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda init <<<
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('$HOME/path/to/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "$HOME/path/to/anaconda3/etc/profile.d/conda.sh"
else
export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
conda activate base
eval $2
Then you need to set the MATLAB_SHELL environment variable either before running MATLAB or in MATLAB itself. The best thing in my opinion would be to do it also in the startup.m file like that:
setenv("MATLAB_SHELL", "/path/to/matlab_shell.sh");
Afterwards you can use the system(...) function to run conda python with all your modules installed like that...
String notation:
system("python -c ""python code goes here"");
Char notation:
system('python -c "python code goes here"');
Hope this helps!
Firstly, if you execute your Python script like a regular system command ([status, commandOut] = system('...python.exe test.py'))
the pyversion (and pyenv, since R2019b) got no effect at all. It only matters if you utilize the py. integration, as in the code below (and, in most cases, this is a way better approach).
Currently (I use R2019b update 5) there's a number of pitfalls, that might cause issues similar to yours. I'd recommend to start from the following:
Create a new clean conda environment:
conda create -n test_py36 python=3.6 numpy
Create the following dummy demo1.py:
def dummy_py_method(x):
return x+1
Create the following run_py_code.m:
function run_py_code()
% explicit module import sometimes show more detailed error messages
py.importlib.import_module('numpy');
% to reloads if there would be any changes:
pymodule = py.importlib.import_module('demo1');
py.importlib.reload(pymodule);
% passing data back and forth
x = rand([3 3]);
x_np = py.numpy.array(x);
y_np=pymodule.dummy_py_method(x_np);
y = double(y_np);
disp(y-x);
Create the following before_first_run.m:
setenv('PYTHONUNBUFFERED','1');
setenv('path',['C:\Users\username\Anaconda3\envs\test_py36\Library\bin;'...
getenv('path')]);
pe=pyenv('Version','C:\users\username\Anaconda3\envs\test_py36\pythonw.exe',...
'ExecutionMode','InProcess'...
);
% add "demo1.py" to path
py_file_path = 'W:\tests\Matlab\python_demos\call_pycode\pycode';
if count(py.sys.path,py_file_path) == 0
insert(py.sys.path,int32(0),py_file_path);
end
Run the before_first_run.m first and run the run_py_code.m next.
Notes:
As already mentioned in this answer, one key point is to add the folder, containing the necessary dll files to the %PATH%, before starting python. This could be achieved with setenv from withing Matlab. Usually, the Library\bin is what should be added.
It might be a good idea to try clean officially-supported CPython distribution (e.g. CPython 3.6.8 ). Only install numpy (python -m pip install numpy). To my experience, the setenv is not necessary in this case.
For me, OutOfProcess mode proved to be buggy. Thus, I'd recommend to explicitly setting InProcess mode (for versions before R2019b, the OutOfProcess option is not present, as well as pyenv).
Do not concatenate the two .m files above into one - the py.importlib statements seem to be pre-executed and thus conflict with pyenv.

MATLAB Engine for Python: Desktop not supported, Swing is not currently available

I need to start MATLAB in Python using MATLAB Engine API for Python as www-data with GUI/as desktop app. And the following code results in the error:
import matlab.engine
eng = matlab.engine.start_matlab()
eng.desktop(nargout=0)
When I try to run this code, the -desktop argument is just ignored and Matlab runs like -nodesktop.
import matlab.engine
eng = matlab.engine.start_matlab('-desktop')
Error: desktop is not supported because: Swing is not currently available.
How to make it run?