passing data from a dataset into a neural network - neural-network

I am an absolute beginner in neural networks and I have problems passing data into the network.
This is the model:
model = keras.Sequential([keras.layers.Flatten(input_shape=(300,150,3)),keras.layers.Dense(128,activation='relu'),keras.layers.Dense(10,activation='softmax')])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
The data consists of images with labels as tuples. Its a mutli-label dataset, where each label, e.g. varroa_output, contains 1 if the characterisitic was present in the image and a 0 if it wasn't. This is the train_batch I tried to feed into the neural network:
<BatchDataset element_spec=(TensorSpec(shape=(None, 300, 150, 3), dtype=tf.uint8, name=None), {'cooling_output': TensorSpec(shape=(None,), dtype=tf.float64, name=None), 'pollen_output': TensorSpec(shape=(None,), dtype=tf.float64, name=None), 'varroa_output': TensorSpec(shape=(None,), dtype=tf.float64, name=None), 'wasps_output': TensorSpec(shape=(None,), dtype=tf.float64, name=None)})>
This is how I tried to train the model:
history = model.fit(train_batches,epochs=5,validation_data=validation_batches)
acc = history.history['accuracy']
print(acc)
And the following error occurs:
ValueError: Found unexpected losses or metrics that do not correspond to any Model output: dict_keys(['cooling_output', 'pollen_output', 'varroa_output', 'wasps_output']). Valid mode output names: ['dense_11']. Received struct is: {'cooling_output': <tf.Tensor 'IteratorGetNext:1' shape=(None,) dtype=float64>, 'pollen_output': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=float64>, 'varroa_output': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=float64>, 'wasps_output': <tf.Tensor 'IteratorGetNext:4' shape=(None,) dtype=float64>}.
I am sure there is an obvious explanation but I am an absolute beginner and would appreciate any help.

Related

how to do Merge Sequential in tf.keras get "A `Concatenate` layer should be called on a list of at least 2 inputs"

I am trying to merge in tf.keras . What is the equivalent.
model = tf.keras.Sequential()
model.add(tf.keras.layers.Concatenate()([sentrnn, qrnn]))
getting following exception
A Concatenate layer should be called on a list of at least 2 inputs
trying to convert this code I found on one of the sites
model.add(Merge([sentrnn, qrnn], mode='concat'))
model.add(Merge([sentrnn, qrnn], mode='sum'))
sentrnn = Sequential()
sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, mask_zero=True))
sentrnn.add(RNN(SENT_HIDDEN_SIZE, return_sequences=False))
sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, input_length=story_maxlen, mask_zero=True))
sentrnn.add(Dropout(0.3))
qrnn = Sequential()
qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE))
qrnn.add(RNN(QUERY_HIDDEN_SIZE, return_sequences=False))
qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, input_length=query_maxlen))
qrnn.add(Dropout(0.3))
qrnn.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
qrnn.add(RepeatVector(story_maxlen))
model = Sequential()
model.add(Merge([sentrnn, qrnn], mode='concat'))
model.add(Merge([sentrnn, qrnn], mode='sum'))
model.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
model.add(Dropout(0.3))
model.add(Dense(vocab_size, activation='softmax'))

How to reproduce a linear regression done via pseudo inverse in pytorch

I try to reproduce a simple linear regression x = A†b using pytorch. But I get completely different numbers.
So first I use plain numpy and do
A_pinv = np.linalg.pinv(A)
betas = A_pinv.dot(b)
print(((b - A.dot(betas))**2).mean())
print(betas)
which results in:
364.12875
[0.43196774 0.14436531 0.42414093]
Now I try to get similar enough numbers using pytorch:
# re-implement via pytoch model using built-ins
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader
# We'll create a TensorDataset, which allows access to rows from inputs and targets as tuples.
# We'll also create a DataLoader, to split the data into batches while training.
# It also provides other utilities like shuffling and sampling.
inputs = to.from_numpy(A)
targets = to.from_numpy(b)
train_ds = TensorDataset(inputs, targets)
batch_size = 5
train_dl = DataLoader(train_ds, batch_size, shuffle=True)
# define model, loss and optimizer
new_model = nn.Linear(source_variables, predict_variables, bias=False)
loss_fn = F.mse_loss
opt = to.optim.SGD(new_model.parameters(), lr=1e-10)
def fit(num_epochs, new_model, loss_fn, opt):
for epoch in tnrange(num_epochs, desc="epoch"):
for xb,yb in train_dl:
# Generate predictions
pred = new_model(xb)
loss = loss_fn(pred, yb)
# Perform gradient descent
loss.backward()
opt.step()
opt.zero_grad()
if epoch % 1000 == 0:
print((new_model.weight, loss))
print('Training loss: ', loss_fn(model(inputs), targets))
# fit the model
fit(10000, new_model, loss_fn, opt)
It prints as the last result:
tensor([[0.0231, 0.5185, 0.4589]], requires_grad=True), tensor(271.8525, grad_fn=<MseLossBackward>))
Training loss: tensor(378.2871, grad_fn=<MseLossBackward>)
As you can see these numbers are completely different so I must have made a mistake somewhere ...
Here are the numbers for A and b to reproduce the result:
A = np.array([[2822.48, 2808.48, 2810.92],
[2832.94, 2822.48, 2808.48],
[2832.57, 2832.94, 2822.48],
[2824.23, 2832.57, 2832.94],
[2854.88, 2824.23, 2832.57],
[2800.71, 2854.88, 2824.23],
[2798.36, 2800.71, 2854.88],
[2818.46, 2798.36, 2800.71],
[2805.37, 2818.46, 2798.36],
[2815.44, 2805.37, 2818.46]], dtype=float32)
b = np.array([2832.94, 2832.57, 2824.23, 2854.88, 2800.71, 2798.36, 2818.46, 2805.37, 2815.44, 2834.4 ], dtype=float32)

Keras does not mach model with classes

I am new to Keras and I am trying to make a Neuronal Network to recognize 38 cases. I created such a model, but it just does not work. There is some problem with last layer I think. I checked summary and it looks like output of last layers is 38 as it should. Can someone help me with making it work?
My code is:
model = Sequential()
model.add(Convolution2D(16, 5, 5, border_mode='valid', input_shape=(168, 192, 3)) )
model.add( Activation('relu') )
model.add( MaxPooling2D(2,2) )
model.add( Convolution2D(16, 5, 5) )
model.add( Activation('relu') )
model.add( MaxPooling2D(2,2) )
model.add( Flatten() )
model.add( Dense(512, activation='relu'))
model.add(Dense(38, activation='softmax'))
model.compile(loss='categorical_crossentropy',optimizer=adam(0.001),metrics=['accuracy'])
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
'data/train',
target_size=(168, 192),
batch_size=38,
class_mode='binary')
validation_generator = test_datagen.flow_from_directory(
'data/validation',
target_size=(168, 192),
batch_size=38,
class_mode='binary')
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=10,
validation_data=validation_generator,
validation_steps=800)
and the error looks like:
ValueError: Error when checking target: expected dense_129 to have shape (None, 38) but got array with shape (38, 1)
According to Keras documentation of from_from_directory, the specified directory ('data/train' in your case) should contain one subdirectory per class.
Since the error is saying the model is getting an array of shape (38, 1), this means you do not have 38 folders with data/train. (Note do not confuse that the first 38 dimension is the batch size, which coincidentally you have set it to same as number of classes, but does not have to be).
So you should either reaarange your subfolders into one class per subfolder, or load data manually, and flow from memory.

Keras: What is the correct data format for recurrent networks?

I am trying to build a recurrent network which classifies sequences (multidimensional data streams). I must be missing something, since while running my code:
from keras.models import Sequential
from keras.layers import LSTM, Dropout, Activation
import numpy as np
ils = 10 # input layer size
ilt = 11 # input layer time steps
hls = 12 # hidden layer size
nhl = 2 # number of hidden layers
ols = 1 # output layer size
p = 0.2 # dropout probability
f_a = 'relu' # activation function
opt = 'rmsprop' # optimizing function
#
# Building the model
#
model = Sequential()
# The input layer
model.add(LSTM(hls, input_shape=(ilt, ils), return_sequences=True))
model.add(Activation(f_a))
model.add(Dropout(p))
# Hidden layers
for i in range(nhl - 1):
model.add(LSTM(hls, return_sequences=True))
model.add(Activation(f_a))
model.add(Dropout(p))
# Output layer
model.add(LSTM(ols, return_sequences=False))
model.add(Activation('softmax'))
model.compile(optimizer=opt, loss='binary_crossentropy')
#
# Making test data and fitting the model
#
m_train, n_class = 1000, 2
data = np.array(np.random.random((m_train, ilt, ils)))
labels = np.random.randint(n_class, size=(m_train, 1))
model.fit(data, labels, nb_epoch=10, batch_size=32)
I get output (truncated):
Using Theano backend.
line 611, in __call__
node = self.make_node(*inputs, **kwargs)
File "/home/koala/.local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 430, in make_node
new_inputs.append(format(outer_seq, as_var=inner_seq))
File "/home/koala/.local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 422, in format
rval = tmp.filter_variable(rval)
File "/home/koala/.local/lib/python2.7/site-packages/theano/tensor/type.py", line 233, in filter_variable
self=self))
TypeError: Cannot convert Type TensorType(float32, 3D) (of Variable Subtensor{:int64:}.0) into Type TensorType(float32, (False, False, True)). You can try to manually convert Subtensor{:int64:}.0 into a TensorType(float32, (False, False, True)).
Is this a problem with the data format at all.
For me the problem was fixed when I went and tried it on my real dataset. The difference being that in the real dataset I have more than 1 label. So an example of dataset on which this code works is:
(...)
ols = 2 # Output layer size
(...)
m_train, n_class = 1000, ols
data = np.array(np.random.random((m_train, ilt, ils)))
labels = np.random.randint(n_class, size=(m_train, 1))
# Make labels onehot
onehot_labels = np.zeros(shape=(labels.shape[0], ols))
onehot_labels[np.arange(labels.shape[0]), labels.astype(np.int)] = 1

Average pooling with Theano

I am trying to implement another pooling function for neural network with Theano, expect of already existing maxpool, for example average pool.
Using to this source, where average pooling is already implemented, my code looks like:
Random initialization just to test:
invals = numpy.random.RandomState(1).rand(3,2,5,5)
Definition of Theano scalars and functions:
pdim = T.scalar('pool dim', dtype='float32')
pool_inp = T.tensor4('pool input', dtype='float32')
pool_sum = TSN.images2neibs(pool_inp, (pdim, pdim))
pool_out = pool_sum.mean(axis=-1)
pool_fun = theano.function([pool_inp, pdim], pool_out, name = 'pool_fun', allow_input_downcast=True)
TSN is theano.sandbox.neighbours
And the call of the function:
pool_dim = 2
temp = pool_fun(invals, pool_dim)
temp.shape = (invals.shape[0], invals.shape[1], invals.shape[2]/pool_dim,
invals.shape[3]/pool_dim)
print ('invals[1,0,:,:]=\n', invals[1,0,:,:])
print ('output[1,0,:,:]=\n',temp[1,0,:,:])
And I am getting an error:
TypeError: neib_shape[0]=2, neib_step[0]=2 and ten4.shape[2]=5 not consistent
Apply node that caused the error: Images2Neibs{valid}(pool input, MakeVector.0, MakeVector.0)
Inputs shapes: [(3, 2, 5, 5), (2,), (2,)]
Inputs strides: [(200, 100, 20, 4), (4,), (4,)]
Inputs types: [TensorType(float32, 4D), TensorType(float32, vector), TensorType(float32, vector)]
Use the Theano flag 'exception_verbosity=high' for a debugprint of this apply node.
I don't really understand this error. Would be glad to have any suggestions how to correct this error or example of other pooling techniques, programmed in Theano.
Thanks!
Edit: with the ignoring the border, it works perfectly
pool_sum = TSN.images2neibs(pool_inp, (pdim, pdim), mode='ignore_borders')
invals[1,0,:,:]=
[[ 0.01936696 0.67883553 0.21162812 0.26554666 0.49157316]
[ 0.05336255 0.57411761 0.14672857 0.58930554 0.69975836]
[ 0.10233443 0.41405599 0.69440016 0.41417927 0.04995346]
[ 0.53589641 0.66379465 0.51488911 0.94459476 0.58655504]
[ 0.90340192 0.1374747 0.13927635 0.80739129 0.39767684]]
output[1,0,:,:]=
[[ 0.33142066 0.30330223]
[ 0.42902038 0.64201581]]
invals has shape (5, 5) in the last two dimensions, however you want to pool over (2, 2) subsets. This only works if you ignore the border (i.e. the last column and the last row of invals).