Cannot add images to an existing classifier - ibm-cloud

I have successfully created a classifier using 20 classes, with 200 images in each class. When I try to add additional classes to this classifier, the API call fails:
Traceback (most recent call last):
File "create_classifier.py", line 34, in <module>
print visual_recognition.update_classifier('flowers_1337689264', **myhash)
File "/usr/local/lib/python2.7/dist-packages/watson_developer_cloud/visual_recognition_v3.py", line 96, in update_classifier
params=params, accept_json=True)
File "/usr/local/lib/python2.7/dist-packages/watson_developer_cloud/watson_developer_cloud_service.py", line 268, in request
raise WatsonException(error_message)
watson_developer_cloud.watson_developer_cloud_service.WatsonException: Unknown error
Here is my code:
import json
from watson_developer_cloud import VisualRecognitionV3
from glob import glob
visual_recognition = VisualRecognitionV3('2016-05-20', api_key='xxxx')
myhash = {}
for f in sorted(glob('*.zip'))[20:30]:
clas = f.replace('.zip','');
clas = clas.replace(' ', '-')
clas += '_positive_examples'
myhash[clas] = open(f, 'rb')
print myhash
#print visual_recognition.create_classifier('test', **myhash)
print visual_recognition.update_classifier('test_1337689264', **myhash)

Not sure if this is your problem, but there is an issue for retraining - If your training .zip file included files that are named like images (for example, *.jpg or *.png), but are not actually valid image files, the system will think it has incomplete data when trying to marshal all the vectors for retraining. This can occur frequently when using .zip files from MacOS, which might include a "hidden" __MACOSX folder with files named like this.
A solution for this issue of non-image files causing retraining to fail was deployed on Aug 10, 2016. Classifiers created after that date should not be hindered in retraining from non-image files presence in the training examples. It's not retroactive, though, so the workaround for classifiers trained before that date is to start the training process over with the same input zip files to create a new one, and then delete the old classifier.

Related

An error occurs when using MLDataTable to load data

I tried to create a word tagger model in Swift according to this tutorial in the latest XCode. But I cannot load data from a local file using MLDataTable. Here is my code.
let data = try MLDataTable(contentsOf:
URL(fileURLWithPath: "/path/to/data.json"))
The error is as follows.
error: Couldn't lookup symbols:
CreateML.MLDataTable.init(contentsOf: Foundation.URL, options:
CreateML.MLDataTable.ParsingOptions) throws -> CreateML.MLDataTable
I tried absolute path and relative path, but neither of them worked(I am pretty sure that the data file is in the right location and the paths are correct). In addition, I can load the local file to a URL object, so the problem should lie in MLDataTable.
Could someone help?
I have the same error however I used .csv file. But the problem is solved when I use COREML tool under developer tools of Xcode.
Here are some recommendations:
Your training data's class column label should be "label"
Your training data can be one file but testing data should contains sub folders named exactly the same name of your label's names. To illustrate, you have "negative", "positive" and "neutral as label names. Then you should have three sub folders named "negative", "positive" and "neutral". Moreover testing data files can't be one json or csv file including all the testing data. For example if you have five rows of negative labeled data, you can't put that csv file under negative sub-folder. You have to create five txt file for each five row.

How does one read multiple DICOM and PNG files at once using pydicom.read_file() and cv2.imread()?

Currently working on a Fully CNN for renal segmentation in MR images. Have 40 images and their ground truth labels, attempting to load all of the images for pre-processing purposes.
Using Google Colab, with the latest versions of pydicom and pip installed, for this project. Currently have the Google Drive mounted to the Colab program and the code below shows the correct pathways to the images and their masks in the pydicom.read_file() and cv2.imread() calls, respectively.
However, when I use the "/../IMG*.dcm" or "/../IMG*.png" file paths (which should be legal?), I receive a "FileNotFoundError" as listed below. But, when I specify a specific .dcm or .png image, the pydicom.read_file() and cv2.imread() calls function quite normally.
Any suggestions on how to resolve this issue? I am struggling a lot with loading the data and pre-processing but have the model architecture ready to go once these preliminary hurdles are overcome.
#import data as data
import pydicom
import numpy as np
images= pydicom.read_file("/content/drive/My Drive/CHOAS_Kidney_Labels/Training_Images/T1DUAL/IMG*.dcm");
numpyArray = images.pixel_array
masks= cv2.imread("/content/drive/My Drive/CHOAS_Kidney_Labels/Ground_Truth_Training/T1DUAL/IMG*.png");
-----> FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/My Drive/CHOAS_Kidney_Labels/Training_Images/T1DUAL/IMG*.dcm'
pydicom.read_file does not support wildcards. You have to iterate over the files yourself, something like (untested):
import glob
import pydicom
pixel_data = []
paths = glob.glob("/content/drive/My Drive/CHOAS_Kidney_Labels/Training_Images/T1DUAL/IMG*.dcm")
for path in paths:
dataset = pydicom.dcmread(path)
pixel_data.append(dataset.pixel_array)

Error when loading .mat file with scipy.io (ValueError: Mat 4 mopt wrong format)

I'm currently trying to load a .mat file in python using scipy and the following bit of code:
from scipy import io as sio
data= "file.mat"
output= sio.loadmat(data)
However when running the command I get the error:
ValueError: Mat 4 mopt wrong format, byteswapping problem?
What does this error message mean? Is there an issue with the file I'm trying to load?
I'm quite the novice when it comes to programming so any suggestions would be greatly appreciated : ) If there is a better way to load .mat files in python I'm open to hearing those too. Thanks in advance!
I've never seen this error before, but it is produced by line 113 in scipy/scipy/io/matlab/mio4.py
def read_header(self):
''' Read and return header for variable '''
data = read_dtype(self.mat_stream, self.dtypes['header'])
name = self.mat_stream.read(int(data['namlen'])).strip(b'\x00')
if data['mopt'] < 0 or data['mopt'] > 5000:
raise ValueError('Mat 4 mopt wrong format, byteswapping problem?')
...
Normally loadmat is the right file loader, at least among the supported types:
v4 (Level 1.0), v6 and v7 to 7.2 matfiles are supported.
Do you know anything about how this file was saved in MATLAB? Any format specifications such as these?

Xmlrpclib and Python DOM marshal

I'm working with Python using the standard xmlrpclib library communicating between a JavaServer (that host xmlrpc server) and my python client. I have a problem and I hope that you should solve my issue.
I want to send a DOM instance through xmlrpc protocol. I know that the instance are impossible to manage by xmlrpclib library in Python but this is what I need. I'm trying to understand why is not possible to marshal the class Nodelist and I recognize that the target function is dumps(self, values) in class Marshaller because she couldn't find this type of Python Object.
For Instance, the Fault code is this:
Traceback (most recent call last):
File "testRestSwi.py", line 31, in <module>
conn.xxx.xxx(dom3,"xxx","xxx")
File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1572, in __request
allow_none=self.__allow_none)
File "/usr/lib/python2.7/xmlrpclib.py", line 1085, in dumps
data = m.dumps(params)
File "/usr/lib/python2.7/xmlrpclib.py", line 632, in dumps
dump(v, write)
File "/usr/lib/python2.7/xmlrpclib.py", line 654, in __dump
f(self, value, write)
File "/usr/lib/python2.7/xmlrpclib.py", line 756, in dump_instance
self.dump_struct(value.__dict__, write)
File "/usr/lib/python2.7/xmlrpclib.py", line 735, in dump_struct
dump(v, write)
File "/usr/lib/python2.7/xmlrpclib.py", line 646, in __dump
raise TypeError, "cannot marshal %s objects" % type(value)
TypeError: cannot marshal <class 'xml.dom.minicompat.NodeList'> objects
Now, I would want to solve this problem and for sure there are many solutions. But I don't know how to develop or implement something within the xmlrpclib in order to avoid the "marshalling" problem. Keep in mind that the file object must be a DOM, is aim, unchangeable.
For Instance, I'm developing the following:
import xml.dom.minidom as parser
import xmlrpclib
dom3 = parser.parseString("xxx")
conn = xmlrpclib.ServerProxy('xxx',allow_none=True,verbose=True)
conn.xxx.xxx(dom3,"xxx","xxx") #<--- The problem Here.
Might you help me?

Celery Pickiling error with dynamic models

I'm using celery with a function which writes data into a database table.
This table doesn't have a related model inside models.py because I create it dynamically using django-mutant.
When I run my task, it correctly writes on my dynamic table, but at the very end of the task I get the following error:
[2013-07-10 09:10:45,707: CRITICAL/MainProcess] Task topology_py.Functions.Functions.objectAggregate[aff70510-1e93-4610-b08c-c3675c92afe9] INTERNAL ERROR: PicklingError("Can't pickle <class 'mutant.apps.tss.models.1_tmp'>: import of module mutant.apps.tss.models failed",)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/task/trace.py", line 261, in trace_task
for callback in task_request.callbacks or []]
File "/usr/lib/python2.7/site-packages/celery/canvas.py", line 197, in apply_async
return self._apply_async(args, kwargs, **options)
File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 472, in apply_async
**options)
File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 249, in publish_task
**kwargs
File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 157, in publish
compression, headers)
File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 233, in _prepare
body) = encode(body, serializer=serializer)
File "/usr/lib/python2.7/site-packages/kombu/serialization.py", line 170, in encode
payload = encoder(data)
File "/usr/lib/python2.7/site-packages/kombu/serialization.py", line 356, in dumps
return dumper(obj, protocol=pickle_protocol)
PicklingError: Can't pickle <class 'mutant.apps.tss.models.1_tmp'>: import of module mutant.apps.tss.models failed
The model that celery is searching for, 1_tmp is not stored in my app, named tss, but inside mutant app tables.
My problem is that if I chain this function as a subtask to another subtask, celery terminates with this error at the end of the first one!
Is there a way to tell celery where to find this model or anyway to skip this error and go further to next subtasks?
Thanks in advance.
EDIT: at the moment I can run even 5 functions in a chain using 5 immutable subtasks. They work correctly and fill correctly the dynamic table, but the first one, and only the first one, continue producing the error described...Can you give me and idea of why only the first one give me the error since that they access the dynamic model in the same identical way?
I resolved: it was an issue inside my first function: I was returning an object of the dynamic model at the end of its execution and that made a celery/django-mutant mess.
This object was completely useless and so deleting its return solved the problem.
Thank you everybody.