I'm trying to run a basic neural net using keras:
import numpy
import pandas
from keras.models import *
from keras.layers import *
from keras.optimizers import *
dataframe = pandas.read_csv("data/mydata.csv", header=None)
dataset = dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:,0:21] #X features
Y = dataset[:,21] # Y labels
#Define Neural network model of 10 Hidden layer with 500 Neurons each
model = Sequential()
model.add(Dense(500, input_dim=21, init='normal', activation='relu')) #Input Layer
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 1
model.add(Dense(500, init='normal', activation='relu')) #Hidden layer 2
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 3
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 4
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 5
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 6
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 7
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 8
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 9
model.add(Dense(500, init='normal', activation='relu')) #Hidden Layer 10
model.add(Dense(1, init='normal', activation='sigmoid')) #Output later
print('Modeled Network')
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'] )
#fits the data point X, Y to the model
model.fit(X, Y, validation_split=0.3, shuffle=True, nb_epoch=50, batch_size=5000)
# serialize model to JSON
print("* Saving model to disk..")
model_json = model.to_json()
with open("models/model.json", "w") as json_file:
json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("models/model.h5")
However, I'm getting the error:
ValueError: could not convert string to float: 'feature21' during compilation. I'm pretty new to this so I'm not sure why it would be trying to convert a string to a float since headers are turned off in the data set and the feature itself is entirely numeric. Is there already a known easy solution to this?
Related
Using a small and imbalanced data set, I put the following codes before all other codes.
from numpy.random import seed
seed(10)
from tensorflow import random
random.set_seed(10)
After standardizing the features, I put
skf = StratifiedKFold(n_splits=10, random_state=10, shuffle=True)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.4, random_state=10)
# define a base ANNC model
def create_model():
model = Sequential()
model.add(Dense(units=64, kernel_initializer='normal', activation='relu', input_shape=(n_features,)))
model.add(Dense(8, kernel_initializer='normal', activation='relu'))
model.add(Dense(4, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics='acc')
return model
After defining a ANN classification, and then I evaluate a base model.
base_model = KerasClassifier(build_fn=create_model, batch_size=10, epochs=1000, verbose=0)
Every time I run the base model, the results are the same. But when I use grid search, the grid_search.best_params_ and grid_search.best_score_ are different. I use the following codes:
# define a ANNC
def create_model():
model = Sequential()
model.add(Dense(units=64, kernel_initializer='normal', activation='relu', input_shape=(n_features,)))
model.add(Dense(8, kernel_initializer='normal', activation='relu'))
model.add(Dense(4, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics='acc')
return model
skf = StratifiedKFold(n_splits=10, random_state=10, shuffle=True)
# create the parameter grid
batch_size = [10,20,30,40,50,60,70,80,90,100]
epochs = [100,200,300,400,500,600,700,800,900,1000]
param_grid = dict(batch_size=batch_size, nb_epoch=epochs)
# create a multi-class ANNC
model = KerasClassifier(build_fn=create_model, verbose=0)
grid_search = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, verbose=0)
# run the GridSearchCV process
grid_search = grid_search.fit(X_train, Y_train)
print("The best values of param_grid are:", grid_search.best_params_, grid_search.best_score_)
I found ANN is very difficult to do when compared to other algorithms. It would be much appreciated if any one could help me. Thanks in advance.
I'm working on a neural network which approximates a function f(X)=y, with X a vector [x0, .., xn] and y in [-inf, +inf]. This approximated function needs to have an accuracy (sum of errors) around 1e-8. In fact, I need my neural network to overfit.
X is composed of random points in the interval -500 and 500. Before putting these points into the input layer I normalized them between [0, 1].
I use keras as follow:
dimension = 10 #example
self.model = Sequential()
self.model.add(Dense(128, input_shape=(dimension,), init='uniform', activation='relu'))
self.model.add(Dropout(.2))
self.model.add(Activation("linear"))
self.model.add(Dense(64, init='uniform', activation='relu'))
self.model.add(Activation("linear"))
self.model.add(Dense(64, init='uniform', activation='relu'))
self.model.add(Dense(1))
X_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))
y_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))
X_scaled = (X_scaler.fit_transform(train_dataset))
y_scaled = (y_scaler.fit_transform(train_labels))
self.model.compile(loss='mse', optimizer='adam')
self.model.fit(X_scaled, y_scaled, epochs=10000, batch_size=10, verbose=1)
I tried different NN, first [n] -> [2] -> [1] with Relu activation function, then [n] -> [128] -> [64] -> [1].
I tried the SGB Optimizer and I slowly increase the learning rate from 1e-9 to 0.1.
I also tried without normalized the data but, in this case, the loss is very high.
My best loss (MSE) is 0.037 with the current setup but i'm far from my goal (1e-8).
First, I would like to know if I did something wrong. I'm in the good way ?
If not, how can I reach my goal ?
Thanks you very much
Try #2
I tried this new configuration:
model = Sequential()
model.add(Dense(128, input_shape=(10,), init='uniform', activation='relu'))
model.add(Dropout(.2))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(64, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))
On a sample of 50 elements, batch_size at 10 and during 100000 epochs.
I get a loss around 1e-4.
Try #3
model.add(Dense(128, input_shape=(10,), activation='tanh'))
model.add(Dense(64, activation='tanh'))
model.add(Dense(1, activation='sigmoid'))
batch_size=1000
epochs=1e5
result: Loss around 1.e-7
I have trained a simple DNN model - using Keras with Theano backend, having:
2 * (3x3 conv layer, 2x2 max-pool, 15% dropout),
96-FC layer,
softmax activation on the final layer,
optimizer: adam.
The goal was to perform a classification into 6 classes.
I have a dataset which consist of 116000 equally distributed among all classes images.
After 14 epochs - the validation results were great:
loss: 0.1, accuracy 92% on a test data - and during the training I had a similar results on my training and validation data.
The task is for learning purpose only - defined in Kaggle website. They want a minimum log-loss result. When I upload my predictions there - the calculated loss is very high ( > 2)
Do you have any suggestions?
some code representing the network and how I use it:
1. Inputs:
116000 images of size 160x160
2. Model
model = Sequential()
model.add(Convolution2D(64, 3, 3, activation='relu', input_shape=(1, 160, 160)))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.15))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.15))
model.add(Flatten())
model.add(Dense(96, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(len(classes), activation='softmax'))
3. Compile and fit:
model.compile(loss='categorical_crossentropy',#categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(np.array(trainData[0]), np.array(trainData[1]),
batch_size=13, nb_epoch=14, verbose=1, shuffle=True, callbacks=callbacks_list, validation_data = (np.array(validationData[0]), np.array(validationData[1])))
4. Test with a new set of samples:
scores = model.evaluate(np.array(testData[0]), np.array(testData[1]), verbose=1)
I am using Keras to build a multi-class (3 classes) image classifier.
I trained the following model with a dataset of approximately 2000 images (1500 training/ 500 validation).
batch_size = 128
nb_classes = 3
nb_epoch = 25
img_rows, img_cols = 128, 128
input_shape = (1, img_rows, img_cols)
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
model = Sequential()
model.add(Convolution2D(32, 5, 5, border_mode='same', input_shape=input_shape))
model.add(Activation('relu'))
model.add(Convolution2D(64, 5, 5, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Dense(nb_classes, activation='softmax'))
lrate = 0.001
decay = lrate/nb_epoch
sgd = SGD(lr=lrate, momentum=0.9, decay=decay, nesterov=False)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
history = model.fit(X_train, Y_train,
batch_size=batch_size,
nb_epoch=nb_epoch,
validation_data=(X_test, Y_test),
shuffle=True,
callbacks=early_stopping)
These are the figures for the training/validation accuracy and loss.
I achieve a 95% training accuracy, 92% validation accuracy and 94% on another separate chunk of images (aside from the 2000 of the dataset) that I have.
Therefore, the model seems to classify reasonably well. However, my problem is that the predicted probabilities for an input image (obtained with the function predict_proba()) are always either 1.0 or 0.0.
Likewise, if I give as an input an image that doesn't belong to any of the 3 classes I would expect some low probabilities (perhaps higher in the most similar class) but I still get 1.0 in one of the classes and 0.0 in the others.
What could be causing that? It seems to me that there is no over fitting. Is there any issue with the model?
Could it be that the images of each class are quite similar between them so somehow the model is quickly too confident on its decision?
I am new to keras and I am trying to built my own neural network.
A task:
I need to write a system that can make decisions for the character, which may meet one or more enemies. The system can be known:
Percentage Health character
Presence of the pistol;
The number of enemies.
The answer must be in the form of one of the following:
Attack
Run
Hide (for a surprise attack)
To do nothing
To train up I made a table of "lessons":
https://i.stack.imgur.com/lD0WX.png
So here is my code:
# Create first network with Keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
import numpy
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# split into input (X) and output (Y) variables
X = numpy.array([[0.5,1,1], [0.9,1,2], [0.8,0,1], [0.3,1,1], [0.6,1,2], [0.4,0,1], [0.9,1,7], [0.5,1,4], [0.1,0,1], [0.6,1,0], [1,0,0]])
Y = numpy.array([[1],[1],[1],[2],[2],[2],[3],[3],[3],[4],[4]])
# create model
model = Sequential()
model.add(Dense(3, input_dim=3, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))
# Compile model
sgd = SGD(lr=0.001)
model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])
# Fit the model
model.fit(X, Y, nb_epoch=150)
# calculate predictions
predictions = model.predict(X)
# round predictions
rounded = [round(x) for x in predictions]
print(rounded)
Here the predictions I get.
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
The accuracy on each epoch is 0.2727 and the loss is decrease.
It's not right.
I was trying to devide learning rate by 10, changing activations and optimizers. Even data I input manually.
Can anyone tell me how to solve my simple problem. thx.
There are several problems in your code.
Number of data entries are very small compared to the NN model.
Y is represented as classes number and not as class vector. A regression model can be learnt on this but its a poor design choice.
output of softmax function is always between 0-1 .. as this is used your model only knows to spew out values between 0-1.
Here below is a bit better modified code:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
import numpy
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# split into input (X) and output (Y) variables
X = numpy.array([[0.5,1,1], [0.9,1,2], [0.8,0,1], [0.3,1,1], [0.6,1,2], [0.4,0,1], [0.9,1,7], [0.5,1,4], [0.1,0,1], [0.6,1,0], [1,0,0]])
y = numpy.array([[1],[1],[1],[2],[2],[2],[3],[3],[3],[0],[0]])
from keras.utils import np_utils
Y = np_utils.to_categorical(y, 4)
# print Y
# create model
model = Sequential()
model.add(Dense(3, input_dim=3, activation='relu'))
model.add(Dense(4, activation='softmax'))
# Compile model
# sgd = SGD(lr=0.1)
# model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
# Fit the model
model.fit(X, Y, nb_epoch=700)
# calculate predictions
predictions = model.predict(X)
predictions_class = predictions.argmax(axis=-1)
print(predictions_class)
Note I have used the softmax activation as the classes are mutually exclusive