I recently switched from MATLAB to Python for data analysis and I am using matplotlib for visualization of the data. This works fine if the number of data points I would like to visualise are low. However, if I would like to visualize e.g.
import matplotlib.pyplot as plt
signal = [round(random.random() * 100) for i in xrange(0, 1000000)]
plt.plot(signal)
plt.show()
I am getting an error:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1489, in __call__
return self.func(*args)
File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 276, in resize
self.show()
File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 348, in draw
FigureCanvasAgg.draw(self)
File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_agg.py", line 451, in draw
self.figure.draw(self.renderer)
File "/usr/lib/pymodules/python2.7/matplotlib/artist.py", line 55, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/pymodules/python2.7/matplotlib/figure.py", line 1034, in draw
func(*args)
File "/usr/lib/pymodules/python2.7/matplotlib/artist.py", line 55, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/pymodules/python2.7/matplotlib/axes.py", line 2086, in draw
a.draw(renderer)
File "/usr/lib/pymodules/python2.7/matplotlib/artist.py", line 55, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/usr/lib/pymodules/python2.7/matplotlib/lines.py", line 562, in draw
drawFunc(renderer, gc, tpath, affine.frozen())
File "/usr/lib/pymodules/python2.7/matplotlib/lines.py", line 938, in _draw_lines
self._lineFunc(renderer, gc, path, trans)
File "/usr/lib/pymodules/python2.7/matplotlib/lines.py", line 978, in _draw_solid
renderer.draw_path(gc, path, trans)
File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_agg.py", line 145, in draw_path
self._renderer.draw_path(gc, path, transform, rgbFace)
OverflowError: Allocated too many blocks
Can you give me an advise what you would do in this case? Are you down sampling your data?
If I perform the same plot in MATLAB even with more data points, I haven't had this problem before.
Not sure exactly what you are trying to show but you can change the plot to '.' and it will work.
import random
import pylab as plt
signal = [round(random.random() * 100) for i in xrange(0, 1000000)]
plt.plot(signal, '.')
plt.show()
You may be able to get what you want by customising using the matlotlibrc file, the docs are here
I think scatter plot is faster
import matplotlib.pyplot as plt
x=np.random.normal(0,1,1000000)
y=np.random.normal(0,1,1000000)
plt.scatter(x,y)
Related
I was trying to write an Excel UDF using xlwings to return finance options calculation from the Mibian library. I've tried the code below.
import xlwings as xw
import mibian
#xw.func
def BSPutOptionImpVol(underlyingPrice,strike,interestRate,expiryDays,premium):
c = mibian.BS([underlyingPrice, strike, interestRate, expiryDays], putPrice=premium)
return c.impliedVolatility
From Excel, I then call the function with the following =BSPutOptionImpVol(45,32,1,127,0.95)
It's returning the following error:
"NameError: name 'norm' is not defined
call = self.underlyingPrice * norm.cdf(self.d1) - \
File ""C:\Users...\anaconda3\lib\site-packages\mibian__init__.py"", line 307, in _price
[self.callPrice, self.putPrice] = self._price()
File ""C:\Users...\anaconda3\lib\site-packages\mibian__init__.py"", line 276, in init
estimate = eval(className)(args, volatility=mid, performance=True).putPrice
File ""C:\Users...\anaconda3\lib\site-packages\mibian__init__.py"", line 29, in impliedVolatility
self.class.name, args, putPrice=self.putPrice)
File ""C:\Users...\anaconda3\lib\site-packages\mibian__init__.py"", line 293, in init
c = mibian.BS([underlyingPrice, strike, interestRate, expiryDays], putPrice=premium)
File ""c:\users...\documents\python scripts\BSPutImVol.py"", line 6, in BSPutOptionImpVol
ret = func(*args)
File ""C:\Users...\anaconda3\lib\site-packages\xlwings\udfs.py"", line 298, in call_udf
res = call_udf(script, fname, args, this_workbook, FromVariant(caller))
File ""C:\Users...\anaconda3\lib\site-packages\xlwings\server.py"", line 195, in CallUDF
return func(args)
File ""C:\Users...\anaconda3\lib\site-packages\win32com\server\policy.py"", line 586, in _invokeex_
return S_OK, -1, self._invokeex_(dispid, lcid, wFlags, args, None, None)
File ""C:\Users...\anaconda3\lib\site-packages\win32com\server\policy.py"", line 283, in _invoke_
return self._invoke_(dispid, lcid, wFlags, args)
File ""C:\Users...\anaconda3\lib\site-packages\win32com\server\policy.py"", line 278, in _Invoke_"
I have also tried just calling the function without passing the parameters in (ie the input values are in the python code) but I still get the same error.
However, if I comment out xlwings and just run the Python code from Spyder as below, it works.
#import xlwings as xw
import mibian
##xw.func
def BSPutOptionImpVol(underlyingPrice,strike,interestRate,expiryDays,premium):
c = mibian.BS([underlyingPrice, strike, interestRate, expiryDays], putPrice=premium)
# return c.impliedVolatility
print(c.impliedVolatility)
I'm a newbie to Python, so appreciate any help and advice. Thanks.
Try..
from scipy.stats import norm
Try add
import scipy
to your code. That resolved the 'norm' issue for me.
This worked for me
pip uninstall numpy scipy
and then
pip install -U numpy scipy
Trying to load a ResNet 18 pre-trained model using the torch.load(PATH) but getting Unicode decode error please help.
Traceback (most recent call last):
File "main.py", line 312, in <module>
main()
File "main.py", line 138, in main
checkpoint = torch.load(args.resume)
File "F:\InsSoft\Anaconda\lib\site-packages\torch\serialization.py", line 593, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "F:\InsSoft\Anaconda\lib\site-packages\torch\serialization.py", line 773, in _legacy_load
result = unpickler.load()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 2: invalid start byte
This error hits whenever the model is pretrained on torch version < 0.4 but using torch version > 0.4 for testing / resuming.
so use checkpoint = torch.load(args.resume,encoding='latin1')
For a deep learning project, I reorganize data from hdf5 files into a big training_data.hfd5 containing batch0...batch_max
When running it in local on my laptop it works, but the process is a bit long (much data), so I tried to run it on big-CPU servers I have access to.
Now I have this error :
Traceback (most recent call last):
File "prepare_data.py", line 88, in <module>
main(readargs())
File "prepare_data.py", line 82, in main
prepare_data('', args)
File "prepare_data.py", line 72, in prepare_data
args.preprocessdatadir, args.datadir, stdout=None)
File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 113, in group_batchs
curr_batch_idx, count_events = organizer.iter_data()
File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 93, in iter_data
self.next_batch(batchsize)
File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 65, in next_batch
self.next_event(fileout)
File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 82, in next_event
fileout[self.group_name + '/label'][self.event_idx] = label
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2798)
File "/home/bizeul/virtual_cpu/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 630, in __setitem__
self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2798)
File "h5py/h5d.pyx", line 221, in h5py.h5d.DatasetID.write (/home/bizeul/virtual_cpu/build/h5py/h5py/h5d.c:3680)
File "h5py/_proxy.pyx", line 132, in h5py._proxy.dset_rw (/home/bizeul/virtual_cpu/build/h5py/h5py/_proxy.c:2022)
File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite (/home/bizeul/virtual_cpu/build/h5py/h5py/_proxy.c:1732)
IOError: Can't write data (No appropriate function for conversion path)
What do you think is the problem ?
Had the same issue. I was trying to create a dataset with set of images and labels. But didn't resize it. After resizing all the images to same size, i was able to run the code and create dataset.
img = cv2.resize(img,(500,500))
Resize after you read the image
I have trained a network model and saved its weights and architecture via checkpoint = ModelCheckpoint(filepath='weights.hdf5') callback. During training, I am using multiple GPUs by calling the funtion below:
def make_parallel(model, gpu_count):
def get_slice(data, idx, parts):
shape = tf.shape(data)
size = tf.concat([ shape[:1] // parts, shape[1:] ],axis=0)
stride = tf.concat([ shape[:1] // parts, shape[1:]*0 ],axis=0)
start = stride * idx
return tf.slice(data, start, size)
outputs_all = []
for i in range(len(model.outputs)):
outputs_all.append([])
#Place a copy of the model on each GPU, each getting a slice of the batch
for i in range(gpu_count):
with tf.device('/gpu:%d' % i):
with tf.name_scope('tower_%d' % i) as scope:
inputs = []
#Slice each input into a piece for processing on this GPU
for x in model.inputs:
input_shape = tuple(x.get_shape().as_list())[1:]
slice_n = Lambda(get_slice, output_shape=input_shape, arguments={'idx':i,'parts':gpu_count})(x)
inputs.append(slice_n)
outputs = model(inputs)
if not isinstance(outputs, list):
outputs = [outputs]
#Save all the outputs for merging back together later
for l in range(len(outputs)):
outputs_all[l].append(outputs[l])
# merge outputs on CPU
with tf.device('/cpu:0'):
merged = []
for outputs in outputs_all:
merged.append(merge(outputs, mode='concat', concat_axis=0))
return Model(input=model.inputs, output=merged)
With the testing code:
from keras.models import Model, load_model
import numpy as np
import tensorflow as tf
model = load_model('cpm_log/deneme.hdf5')
x_test = np.random.randint(0, 255, (1, 368, 368, 3))
output = model.predict(x = x_test, batch_size=1)
print output[4].shape
I got the error below:
Traceback (most recent call last):
File "cpm_test.py", line 5, in <module>
model = load_model('cpm_log/Jun5_1000/deneme.hdf5')
File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 240, in load_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 301, in model_from_config
return layer_module.deserialize(config, custom_objects=custom_objects)
File "/usr/local/lib/python2.7/dist-packages/keras/layers/__init__.py", line 46, in deserialize
printable_module_name='layer')
File "/usr/local/lib/python2.7/dist-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
list(custom_objects.items())))
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2378, in from_config
process_layer(layer_data)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2373, in process_layer
layer(input_tensors[0], **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 578, in __call__
output = self.call(inputs, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/layers/core.py", line 659, in call
return self.function(inputs, **arguments)
File "/home/muhammed/DEV_LIBS/developments/mocap/pose_estimation/training/cpm/multi_gpu.py", line 12, in get_slice
def get_slice(data, idx, parts):
NameError: global name 'tf' is not defined
By inspecting the error output, I decide that the problem is with the parallelization code. However, I can't resolve the issue.
You may need to use custom_objects to enable loading of the model.
import tensorflow as tf
model = load_model('model.h5', custom_objects={'tf': tf,})
I am trying to find the indices 2d max pooling in lasagne
network = batch_norm(Conv2DLayer(
network, num_filters=filter_size, filter_size=(kernel, kernel),pad=pad,
nonlinearity=lasagne.nonlinearities.rectify,
W=lasagne.init.GlorotUniform(),name="conv"), name="BN")
pool_in = lasagne.layers.get_output(network)
network = MaxPool2DLayer(network, pool_size=(pool_size, pool_size),stride=2,name="pool")
pool_out = lasagne.layers.get_output(network)
ind1 = T.grad(T.sum(pool_out), wrt=pool_in)
When I try to build the model it raises a error
DisconnectedInputError: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: Elemwise{mul,no_inplace}.0
Backtrace when the node is created:
File "//anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2871, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "//anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2975, in run_ast_nodes
if self.run_code(code, result):
File "//anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 3035, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-28-0b136cc660e2>", line 1, in <module>
network = build_model()
File "<ipython-input-27-20acc3fe0d98>", line 8, in build_model
pool_in = lasagne.layers.get_output(network)
File "//anaconda/lib/python2.7/site-packages/lasagne/layers/helper.py", line 191, in get_output
all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs)
File "//anaconda/lib/python2.7/site-packages/lasagne/layers/special.py", line 52, in get_output_for
return self.nonlinearity(input)
File "//anaconda/lib/python2.7/site-packages/lasagne/nonlinearities.py", line 157, in rectify
return theano.tensor.nnet.relu(x)
What is the right way of coding functions on lasagne layers intermediate outputs.
I had a similar problem a while ago, check out my solution for 2d and 3d max pooling indices:
Theano max_pool_3d
(Its based on the same Google-groups post, i guess)