dropout_rate and learning_rate do not change in RandomSearchCV - neural-network

Question

Dears, does anyone has an ideia why 'dropout_rate' and 'learning_rate' returne only 0 and does not search for the range I gave when I am doing a RandomizedSearchCV on the hyperparameters?
Here is my code for a ANN using keras/tensoflow:
# Create the model
def create_model(neurons = 1, init_mode = 'uniform', activation='relu', inputDim = 8792, dropout_rate=0.7, learn_rate=0.01, momentum=0, weight_constraint=0): #, learn_rate=0.01, momentum=0):
model = Sequential()
model.add(Dense(neurons, input_dim=inputDim, kernel_initializer=init_mode, activation=activation, kernel_constraint=maxnorm(weight_constraint), kernel_regularizer=regularizers.l2(0.001))) # one inner layer
#model.add(Dense(neurons, input_dim=inputDim, activation=activation)) # second inner layer
model.add(Dropout(dropout_rate))
model.add(Dense(1, activation='sigmoid'))
optimizer = RMSprop(lr=learn_rate)
# compile model
model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
return model
# model
model = KerasClassifier(build_fn=create_model, verbose=0)
# Define K-fold cross validation test harness
kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=seed)
for train, test in kfold.split(X_train, Y_train):
print("TRAIN:", train, "VALIDATION:", test)
# Define Hyperparameters
# specify parameters and distributions to sample from
from scipy.stats import randint as sp_randint
param_dist = {'neurons': sp_randint(300, 360), #, 175, 180, 185, 190, 195, 200],
'learn_rate': sp_randint (0.001, 0.01),
'batch_size': sp_randint(50, 60),
'epochs': sp_randint(20, 30),
'dropout_rate': sp_randint(0.2, 0.8),
'weight_constraint': sp_randint(3, 8)
}
# run randomized search
n_iter_search = 100
print("[INFO] Starting training digits")
print("[INFO] Tuning hyper-parameters for accuracy")
grid = RandomizedSearchCV(estimator=model, param_distributions=param_dist,
n_iter=n_iter_search, n_jobs=10, cv=kfold)
start = time.time()
grid_result = grid.fit(X_train, Y_train)
print("[INFO] GridSearch took {:.2f} seconds".format(time.time() - start))
My answer:
[INFO] GridSearch took 1164.39 seconds
[INFO] GridSearch best score 1.000000 using parameters: {'batch_size': 54, 'dropout_rate': 0, 'epochs': 20, 'learn_rate': 0, 'neurons': 331, 'weight_constraint': 7}
[INFO] Grid scores on development set:
0.614679 (0.034327) with: {'batch_size': 54, 'dropout_rate': 0, 'epochs': 29, 'learn_rate': 0, 'neurons': 354, 'weight_constraint': 6}
0.883792 (0.008650) with: {'batch_size': 53, 'dropout_rate': 0, 'epochs': 27, 'learn_rate': 0, 'neurons': 339, 'weight_constraint': 7}
0.256881 (0.012974) with: {'batch_size': 59, 'dropout_rate': 0, 'epochs': 27, 'learn_rate': 0, 'neurons': 308, 'weight_constraint': 4}
...
Thanks for helping.

Answer 1

0.2 and 0.8 are not integers, so when you use sp_randint(0.2, 0.8), these are converted to integers so its the same as sp_randint(0, 0). You have to use an equivalent function that generates floating point numbers, not integers.
For example, you can use a uniform distribution (uniform from scipy.stats) to generate real numbers.