How to use numba in Colaboratory - numba

Anybody tried to use numba in google collaboratory? I just can not figure out how to set it up in this environment.
At the moment, I'm stuck with the error library nvvm not found.

Copy this code into cell. It works for me.
!apt-get install nvidia-cuda-toolkit
!pip3 install numba
import os
os.environ['NUMBAPRO_LIBDEVICE'] = "/usr/lib/nvidia-cuda-toolkit/libdevice"
os.environ['NUMBAPRO_NVVM'] = "/usr/lib/x86_64-linux-gnu/libnvvm.so"
from numba import cuda
import numpy as np
import time
#cuda.jit
def hello(data):
data[cuda.blockIdx.x, cuda.threadIdx.x] = cuda.blockIdx.x
numBlocks = 5
threadsPerBlock = 10
data = np.ones((numBlocks, threadsPerBlock), dtype=np.uint8)
hello[numBlocks, threadsPerBlock](data)
print(data)

I didn't have to install the packages #Algis suggested, but the paths to the drivers were different. So I had to do the following.
First determine the correct paths for the drivers
!find / -iname 'libdevice'
!find / -iname 'libnvvm.so'
# Output:
# /usr/local/cuda-9.2/nvvm/lib64/libnvvm.so
# /usr/local/cuda-9.2/nvvm/libdevice
Then set the paths as #Algis described
import os
os.environ['NUMBAPRO_LIBDEVICE'] = "/usr/local/cuda-9.2/nvvm/libdevice"
os.environ['NUMBAPRO_NVVM'] = "/usr/local/cuda-9.2/nvvm/lib64/libnvvm.so"

You can do #Stan's work in one simple sweep if you have this block at the beginning of your colab notebook (this also automatically updates as CUDA gets updated)
import os
dev_lib_path = !find / -iname 'libdevice'
nvvm_lib_path = !find / -iname 'libnvvm.so'
assert len(dev_lib_path)>0, "Device Lib Missing"
assert len(nvvm_lib_path)>0, "NVVM Missing"
os.environ['NUMBAPRO_LIBDEVICE'] = dev_lib_path[0]
os.environ['NUMBAPRO_NVVM'] = nvvm_lib_path[0]

Related

How I can overcome the _geoslib problem in this code?

This code is specified to visualize the CALIPSO satellite atmospheric profiles
The input files are .HDF
The code is copyrighted to the HDF group.
In the begining, I struggled with installing the basemap,
finally I installed it using .whl file on my windows10.
Now, this error is reached when I run the script:
SystemError:
execution of module _geoslib raised unreported exception.
I have looked a lot in google, but nothing done.
Can you please help me?
Cheers
"Copyright (C) 2014-2019 The HDF Group
Copyright (C) 2014 John Evans
This example code illustrates how to access and visualize a LaRC CALIPSO file
in file in Python.
If you have any questions, suggestions, or comments on this example, please use
the HDF-EOS Forum (http://hdfeos.org/forums). If you would like to see an
example of any other NASA HDF/HDF-EOS data product that is not listed in the
HDF-EOS Comprehensive Examples page (http://hdfeos.org/zoo), feel free to
contact us at eoshelp#hdfgroup.org or post it at the HDF-EOS Forum
(http://hdfeos.org/forums).
Usage: save this script and run
$python CAL_LID_L2_VFM-ValStage1-V3-02.2011-12-31T23-18-11ZD.hdf.py
The HDF file must either be in your current working directory
or in a directory specified by the environment variable HDFEOS_ZOO_DIR.
Tested under: Python 2.7.15::Anaconda custom (64-bit)
Last updated: 2019-01-25
"""
import os
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.basemap import Basemap
from matplotlib import colors
USE_NETCDF4 = False
def run(FILE_NAME):
# Identify the data field.
DATAFIELD_NAME = 'Feature_Classification_Flags'
if USE_NETCDF4:
from netCDF4 import Dataset
nc = Dataset(FILE_NAME)
# Subset the data to match the size of the swath geolocation fields.
# Turn off autoscaling, we'll handle that ourselves due to presence of
# a valid range.
var = nc.variables[DATAFIELD_NAME]
data = var[:,1256]
# Read geolocation datasets.
lat = nc.variables['Latitude'][:]
lon = nc.variables['Longitude'][:]
else:
from pyhdf.SD import SD, SDC
hdf = SD(FILE_NAME, SDC.READ)
# Read dataset.
data2D = hdf.select(DATAFIELD_NAME)
data = data2D[:,1256]
# Read geolocation datasets.
latitude = hdf.select('Latitude')
lat = latitude[:]
longitude = hdf.select('Longitude')
lon = longitude[:]
# Subset data. Otherwise, all points look black.
lat = lat[::10]
lon = lon[::10]
data = data[::10]
# Extract Feature Type only through bitmask.
data = data & 7
# Make a color map of fixed colors.
cmap = colors.ListedColormap(['black', 'blue', 'yellow', 'green', 'red', 'purple', 'gray', 'white'])
# The data is global, so render in a global projection.
m = Basemap(projection='cyl', resolution='l',
llcrnrlat=-90, urcrnrlat=90,
llcrnrlon=-180, urcrnrlon=180)
m.drawcoastlines(linewidth=0.5)
m.drawparallels(np.arange(-90.,90,45))
m.drawmeridians(np.arange(-180.,180,45), labels=[True,False,False,True])
x,y = m(lon, lat)
i = 0
for feature in data:
m.plot(x[i], y[i], 'o', color=cmap(feature), markersize=3)
i = i+1
long_name = 'Feature Type at Altitude = 2500m'
basename = os.path.basename(FILE_NAME)
plt.title('{0}\n{1}'.format(basename, long_name))
fig = plt.gcf()
# define the bins and normalize
bounds = np.linspace(0,8,9)
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
# create a second axes for the colorbar
ax2 = fig.add_axes([0.93, 0.2, 0.01, 0.6])
cb = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, spacing='proportional', ticks=bounds, boundaries=bounds, format='%1i')
cb.ax.set_yticklabels(['invalid', 'clear', 'cloud', 'aerosol', 'strato', 'surface', 'subsurf', 'no signal'], fontsize=5)
# plt.show()
pngfile = "{0}.py.png".format(basename)
fig.savefig(pngfile)
if __name__ == "__main__":
# If a certain environment variable is set, look there for the input
# file, otherwise look in the current directory.
hdffile = 'CAL_LID_L2_VFM-ValStage1-V3-02.2011-12-31T23-18-11ZD.hdf'
try:
fname = os.path.join(os.environ['HDFEOS_ZOO_DIR'], ncfile)
except KeyError:
fname = hdffile
run(fname)
Please try miniconda and use basemap from conda-forge:
conda install -c conda-forge basemap

Preview app doesn't open since I installed MACOS Catalina

import PIL
img = PIL.Image.new("RGB", (100,100))
img.show()
The error message:
FSPathMakeRef(/Applications/Preview.app) failed with error -43.
Following from Sean True's answer, an even quicker but temporary fix is to simply make a symbolic link to Preview.app in the old location. In the terminal run
ln -s /System/Applications/Preview.app /Applications/Preview.app
This fixed the problem for me.
There's an official fix in github for Pillow 7, but I'm still on 6.
This appears to be a PIL ImageShow issue, with the PIL MacViewer using /Applications/Preview.app as an absolute path to the OSX Preview app.
It's not there in Catalina. I did a quick hack to ImageShow.py changing /Applications/Preview.app to just Preview.app and the issue went away. That might or might not still work on pre-Catalina OSX, but I don't have an easy way to test.
It has apparently moved to /System/Applications/Preview.app so a quick check at run time would probably cover both cases.
elif sys.platform == "darwin":
class MacViewer(Viewer):
format = "PNG"
options = {'compress_level': 1}
preview_locations = ["/System/Applications/Preview.app","/Applications/Preview.app"]
preview_location = None
def get_preview_application(self):
if self.preview_location is None:
for pl in self.preview_locations:
if os.path.exists(pl):
self.preview_location = pl
break
if self.preview_location is None:
raise RuntimeError("Can't find Preview.app in %s" % self.preview_locations)
return self.preview_location
def get_command(self, file, **options):
# on darwin open returns immediately resulting in the temp
# file removal while app is opening
pa = self.get_preview_application()
command = "open -a %s" % pa
command = "(%s %s; sleep 20; rm -f %s)&" % (command, quote(file),
quote(file))
return command
def show_file(self, file, **options):
"""Display given file"""
pa = self.get_preview_application()
fd, path = tempfile.mkstemp()
with os.fdopen(fd, 'w') as f:
f.write(file)
with open(path, "r") as f:
subprocess.Popen([
'im=$(cat);'
'open %s $im;'
'sleep 20;'
'rm -f $im' % pa
], shell=True, stdin=f)
os.remove(path)
return 1

Is possible to add a python path with pipenv?

I have some old python projects that are NOT packaged (no setup.py) I need to access them.
Till now I was using buildout where I can specify a python path in buildout.cfg.
I would like to switch to pipenv : how can I specify an arbitrary python path with pipenv so I can import my unpackaged projects ?
I finally wrote a script that creates a .pth file with all the pythonpath I wanted to add in the virtualenv:
#!/usr/bin/env python
from distutils import sysconfig
import os
site_packages_path = sysconfig.get_python_lib()
PTH_FILENAME = 'MyApp.pth'
# Change here to your project home dir
PROJECT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
relative_paths = [
'.',
'plugins',
'src/achilterm',
'src/addicted',
'src/naghelp',
'src/textops',
'src',
]
absolute_paths = [ os.path.join(PROJECT_DIR,p) for p in relative_paths ]
with open(os.path.join(site_packages_path,PTH_FILENAME),'w') as fh:
for p in absolute_paths:
print 'Installing path : %s ...' % p
fh.write('%s\n' % p)

simpleITK N4BiasFieldCorrection performs strange in terms of running speed and results

I am doing a project about brain tumor segmentation. And when I apply N4BiasCorrection to my file(.mha), I used slicer and simpleITK methods.
Slicer performs well but is time-consuming because I do not know how to use code to run through all my file, I just use the Slicer-N4ITK module and process each file by hand.
Then I try the simpleITK with python, problems show up. First, it runs very slow on each .mha file and gets a really big file(36.7MB compare with 4.4MB using Slicer) after applying n4biasfieldcorrection. Second, in order to speed up, I set the Shrink parameter to 4 but the whole .mha file becomes really blurred, which will not happen using slicer.
So can anyone tell me whether it is normal ? are there any methods to speed up without blurring my file? Or could you please tell me an example to apply N4BiasFieldCorrection within Slicer python interactor .
Thanks!!
# -*- coding: utf-8 -*-
"""
Spyder Editor
This is a temporary script file.
"""
from __future__ import print_function
import SimpleITK as sitk
import sys
import os
#from skimage import io
from glob import glob
import numpy as np
def n4process(inputimage, outpath):
inputImage = sitk.ReadImage( inputimage )
# numberFilltingLevels = 4
maskImage = sitk.OtsuThreshold( inputImage, 0, 1, 200 )
# inputImage = sitk.Shrink( inputImage, [ 2 ] * inputImage.GetDimension() )
# maskImage = sitk.Shrink( maskImage, [ 2 ] * inputImage.GetDimension() )
inputImage = sitk.Cast( inputImage, sitk.sitkFloat32 )
corrector = sitk.N4BiasFieldCorrectionImageFilter();
corrector.SetConvergenceThreshold=0.001
corrector.SetBiasFieldFullWidthAtHalfMaximum=0.15
corrector.SetMaximumNumberOfIterations=50
corrector.SetNumberOfControlPoints=4
corrector.SetNumberOfHistogramBins=200
corrector.SetSplineOrder=3
corrector.SetWienerFilterNoise=0.1
output = corrector.Execute( inputImage,maskImage )
sitk.WriteImage( output, outpath )
input_path = '/Users/chenrui/Desktop/BRATS2015_Training/HGG/'
patientpath = glob('/Users/chenrui/Desktop/BRATS2015_Training/HGG/*')
num = 0
for i in patientpath:
num = num+1
#i = '/Users/chenrui/Desktop/BRATS2015_Training/HGG/brats_2013_pat0001_1'
flair = glob(i + '/*Flair*/*.mha')
flair_outpath = '/Users/chenrui/Desktop/BRATS2015_Training/test/'+'Flair/'+str(num)+'.mha'
n4process(flair[0], flair_outpath)
t2 = glob(i + '/*T2*/*.mha')
t2_outpath = '/Users/chenrui/Desktop/BRATS2015_Training/HGG_n4/'+'T2/'+str(num)+'.mha'
n4process(t2[0], t2_outpath)
t1c = glob(i + '/*_T1c*/*.mha')
t1c_outpath = '/Users/chenrui/Desktop/BRATS2015_Training/HGG_n4/'+'T1c/'+str(num)+'.mha'
n4process(t1c[0], t1c_outpath)
t1 = glob(i + '/*_T1*/*.mha')
t1 = [scan for scan in t1 if scan not in t1c]
t1_outpath = '/Users/chenrui/Desktop/BRATS2015_Training/HGG_n4/'+'T1/'+str(num)+'.mha'
n4process(t1[0],t1_outpath)
From a look at the original implementation http://www.insight-journal.org/browse/publication/640
You can download this and generate the example to then test on your data. The parameters you set appear to be the same as defined in the defaults except for WeinerFilterNoise which should be 0.01 unless you've changed this for a reason - is this the blurring issue?
The size differential (x 8 increase) will be that you've probably saved out the data from 8bit to 64 bit or something. Checking the metaimage header will show this. This can be resolved with casting.

How do you make NLTK draw() trees that are inline in iPython/Jupyter

For Matplotlib plots in iPython/Jupyter you can make the notebook plot plots inline with
%matplotlib inline
How can one do the same for NLTK draw() for trees? Here is the documentation http://www.nltk.org/api/nltk.draw.html
Based on this answer:
import os
from IPython.display import Image, display
from nltk.draw import TreeWidget
from nltk.draw.util import CanvasFrame
def jupyter_draw_nltk_tree(tree):
cf = CanvasFrame()
tc = TreeWidget(cf.canvas(), tree)
tc['node_font'] = 'arial 13 bold'
tc['leaf_font'] = 'arial 14'
tc['node_color'] = '#005990'
tc['leaf_color'] = '#3F8F57'
tc['line_color'] = '#175252'
cf.add_widget(tc, 10, 10)
cf.print_to_file('tmp_tree_output.ps')
cf.destroy()
os.system('convert tmp_tree_output.ps tmp_tree_output.png')
display(Image(filename='tmp_tree_output.png'))
os.system('rm tmp_tree_output.ps tmp_tree_output.png')
Little slow, but does the job. If you're doing it remotely, don't forget to run your ssh session with -X key (like ssh -X user#server.com) so that Tk could initialize itself (no display name and no $DISPLAY environment variable-kind of error)
UPD: it seems like last versions of jupyter and nltk work nicely together, so you can just do IPython.core.display.display(tree) to get a nice-looking tree-render embedded into the output.
2019 Update:
This runs on Jupyter Notebook:
from nltk.tree import Tree
from IPython.display import display
tree = Tree.fromstring('(S (NP this tree) (VP (V is) (AdjP pretty)))')
display(tree)
Requirements:
NLTK
Ghostscript