inconsistent results between Keras and MLPClassifier sklearn - neural-network

Question

I have been reading Keras documentation to build my own MLP network that implements MLP backpropagation. I am familiar with MLPClassifier in sklearn but I want to learn Keras for deep learning. The following is the first attempt. The network has 3 layers of 1 input (features=64), 1 output and 1 hidden. The total is (64,64,1). The input is numpy matrix X of 125K samples (64 dim) and y is a 1D numpy binary class (1, -1):
# Keras imports
from keras.models import Sequential
from sklearn.model_selection import train_test_split
from keras.layers import Dense, Dropout, Activation
from keras.initializers import RandomNormal, VarianceScaling, RandomUniform
from keras.optimizers import SGD, Adam, Nadam, RMSprop
# System imports
import sys
import os
import numpy as np
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
def train_model(X, y, num_streams, num_stages):
'''
STEP1: Initialize the Model
'''
tr_X, ts_X, tr_y, ts_y = train_test_split(X, y, train_size=.8)
model = initialize_model(num_streams, num_stages)
'''
STEP2: Train the Model
'''
model.compile(loss='binary_crossentropy',
optimizer=Adam(lr=1e-3),
metrics=['accuracy'])
model.fit(tr_X, tr_y,
validation_data=(ts_X, ts_y),
epochs=3,
batch_size=200)
def initialize_model(num_streams, num_stages):
model = Sequential()
hidden_units = 2 ** (num_streams + 1)
# init = VarianceScaling(scale=5.0, mode='fan_in', distribution='normal')
init_bound1 = np.sqrt(3.5 / ((num_stages + 1) + num_stages))
init_bound2 = np.sqrt(3.5 / ((num_stages + 1) + hidden_units))
init_bound3 = np.sqrt(3.5 / (hidden_units + 1))
# drop_out = np.random.uniform(0, 1, 3)
# This is the input layer (that's why you have to state input_dim value)
model.add(Dense(num_stages,
input_dim=num_stages,
activation='relu',
kernel_initializer=RandomUniform(minval=-init_bound1, maxval=init_bound1)))
model.add(Dense(hidden_units,
activation='relu',
kernel_initializer=RandomUniform(minval=-init_bound2, maxval=init_bound2)))
# model.add(Dropout(drop_out[1]))
# This is the output layer
model.add(Dense(1,
activation='sigmoid',
kernel_initializer=RandomUniform(minval=-init_bound3, maxval=init_bound3)))
return model
The problem is that I get 99% accuracy with the same dataset X and y when using MLPClassifier sklearn. However, Keras gives poor accuracy as seen below:
Train on 100000 samples, validate on 25000 samples
Epoch 1/3
100000/100000 [==============================] - 1s - loss: -0.5358 - acc: 0.0022 - val_loss: -0.7322 - val_acc: 0.0000e+00
Epoch 2/3
100000/100000 [==============================] - 1s - loss: -0.6353 - acc: 0.0000e+00 - val_loss: -0.7385 - val_acc: 0.0000e+00
Epoch 3/3
100000/100000 [==============================] - 1s - loss: -0.7720 - acc: 9.0000e-05 - val_loss: -0.9474 - val_acc: 5.2000e-04
I don't understand why? Am I missing something here? Any help is appreciated.

Answer 1

I think the problem is that you are using a sigmoid output layer (bound to [0, 1]) but your classes are (1, -1), you need to change your output values or use tanh.
Also keras layers may have different default parameters than sklearn, make sure you take a look at those in the documentation.
One last thing, for the kernel_initializer try glorot_uniform, it is a good default.

Answer 2

check by converting your labeled data to one hot code before training the model.
For more info on why one hot code check out https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/