Trouble with setting the objective function in BayesianOptimization tuner in keras_tuner - tf.keras

Question

I'm trying to tune hyperparameters for an LSTM model in Keras using Keras tuner's BayesianOptimization tuner. I keep getting error messages that seem to object to what I put in the objective argument when I instantiate the tuner. Here's my code:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dropout, Dense
from kerastuner.tuners import BayesianOptimization
from kerastuner.engine.hyperparameters import HyperParameters
from kerastuner import Objective
def build_model(hp):
model = Sequential()
model.add(LSTM(hp.Int("input_unit", min_value=20, max_value=100, step=20),
batch_input_shape=(149,1,1), stateful=True, return_sequences=True))
for i in range(hp.Int("n_layers", 1, 2)):
model.add(LSTM(hp.Int(f"lstm_{i}_units", min_value=20, max_value=100, step=20),
stateful=True, return_sequences=True))
model.add(Dropout(hp.Float("Dropout_rate", min_value=0, max_value=0.5, step=0.1)))
model.add(Dense(1))
model.compile(loss="mean_squared_error",
optimizer=hp.Choice("optimizer", values=["rmsprop", "adam"]),
metrics=[keras.metrics.RootMeanSquaredError(name="rmse")])
return model
tuner = BayesianOptimization(build_model,
objective=Objective("root_mean_squared_error", direction="min"),
max_trials=5,
executions_per_trial=1)
tuner.search(X_train, y_train, epochs=30, batch_size=149, shuffle=False, validation_data=(X_test, y_test))
Below is the message I get back. As you can see, it appears that the code executes at least partially, since I'm getting a message back about what hyperparameters perform best so far, and I get the summary of the epochs. After that, a long line of error messages with the last one seeming to be related to the objective argument. Is the issue with how I am specifying the objective function when I instantiate the tuner, or is there some other issue?
INFO:tensorflow:Reloading Oracle from existing project ./untitled_project/oracle.json
INFO:tensorflow:Reloading Tuner from ./untitled_project/tuner0.json
Search: Running Trial #3
Value |Best Value So Far |Hyperparameter
20 |40 |input_layer
1 |0 |middle_layers
0.1 |0.2 |dropout_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.momentum
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.rho
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'rms' for (root).layer_with_weights-1.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'rms' for (root).layer_with_weights-1.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
Epoch 1/30
29/29 [==============================] - 3s 25ms/step - loss: 0.2129 - rmse: 0.4614 - val_loss: 0.2266 - val_rmse: 0.4761
Epoch 2/30
29/29 [==============================] - 0s 8ms/step - loss: 0.1665 - rmse: 0.4080 - val_loss: 0.2074 - val_rmse: 0.4554
Epoch 3/30
29/29 [==============================] - 0s 7ms/step - loss: 0.1618 - rmse: 0.4022 - val_loss: 0.2002 - val_rmse: 0.4474
Epoch 4/30
29/29 [==============================] - 0s 7ms/step - loss: 0.1575 - rmse: 0.3969 - val_loss: 0.1906 - val_rmse: 0.4366
Epoch 5/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1513 - rmse: 0.3890 - val_loss: 0.1788 - val_rmse: 0.4228
Epoch 6/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1421 - rmse: 0.3770 - val_loss: 0.1638 - val_rmse: 0.4047
Epoch 7/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1283 - rmse: 0.3583 - val_loss: 0.1440 - val_rmse: 0.3794
Epoch 8/30
29/29 [==============================] - 0s 6ms/step - loss: 0.1073 - rmse: 0.3275 - val_loss: 0.1161 - val_rmse: 0.3408
Epoch 9/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0773 - rmse: 0.2781 - val_loss: 0.0807 - val_rmse: 0.2841
Epoch 10/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0509 - rmse: 0.2256 - val_loss: 0.0613 - val_rmse: 0.2475
Epoch 11/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0449 - rmse: 0.2119 - val_loss: 0.0594 - val_rmse: 0.2438
Epoch 12/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0440 - rmse: 0.2098 - val_loss: 0.0587 - val_rmse: 0.2423
Epoch 13/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0434 - rmse: 0.2084 - val_loss: 0.0583 - val_rmse: 0.2414
Epoch 14/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0431 - rmse: 0.2076 - val_loss: 0.0580 - val_rmse: 0.2408
Epoch 15/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0429 - rmse: 0.2070 - val_loss: 0.0578 - val_rmse: 0.2405
Epoch 16/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0427 - rmse: 0.2066 - val_loss: 0.0577 - val_rmse: 0.2402
Epoch 17/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0426 - rmse: 0.2064 - val_loss: 0.0576 - val_rmse: 0.2400
Epoch 18/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0425 - rmse: 0.2062 - val_loss: 0.0575 - val_rmse: 0.2398
Epoch 19/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0424 - rmse: 0.2060 - val_loss: 0.0575 - val_rmse: 0.2397
Epoch 20/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0424 - rmse: 0.2059 - val_loss: 0.0574 - val_rmse: 0.2396
Epoch 21/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0423 - rmse: 0.2058 - val_loss: 0.0573 - val_rmse: 0.2395
Epoch 22/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0423 - rmse: 0.2057 - val_loss: 0.0573 - val_rmse: 0.2394
Epoch 23/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0423 - rmse: 0.2056 - val_loss: 0.0572 - val_rmse: 0.2393
Epoch 24/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2055 - val_loss: 0.0572 - val_rmse: 0.2392
Epoch 25/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2055 - val_loss: 0.0572 - val_rmse: 0.2391
Epoch 26/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2054 - val_loss: 0.0571 - val_rmse: 0.2390
Epoch 27/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2054 - val_loss: 0.0571 - val_rmse: 0.2390
Epoch 28/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0422 - rmse: 0.2053 - val_loss: 0.0571 - val_rmse: 0.2389
Epoch 29/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0421 - rmse: 0.2053 - val_loss: 0.0570 - val_rmse: 0.2388
Epoch 30/30
29/29 [==============================] - 0s 6ms/step - loss: 0.0421 - rmse: 0.2052 - val_loss: 0.0570 - val_rmse: 0.2388
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/tmp/ipykernel_371/3389478107.py in <module>
26 executions_per_trial=1)
27
---> 28 tuner.search(X_train, y_train, epochs=30, batch_size=149, shuffle=False, validation_data=(X_test, y_test))
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/base_tuner.py in search(self, *fit_args, **fit_kwargs)
177
178 self.on_trial_begin(trial)
--> 179 results = self.run_trial(trial, *fit_args, **fit_kwargs)
180 # `results` is None indicates user updated oracle in `run_trial()`.
181 if results is None:
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/tuner.py in run_trial(self, trial, *args, **kwargs)
292 callbacks.append(model_checkpoint)
293 copied_kwargs["callbacks"] = callbacks
--> 294 obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
295
296 histories.append(obj_value)
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/tuner.py in _build_and_fit_model(self, trial, *args, **kwargs)
221 model = self._try_build(hp)
222 results = self.hypermodel.fit(hp, model, *args, **kwargs)
--> 223 return tuner_utils.convert_to_metrics_dict(
224 results, self.oracle.objective, "HyperModel.fit()"
225 )
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/tuner_utils.py in convert_to_metrics_dict(results, objective, func_name)
266 best_epoch = 0
267 for epoch, metrics in epoch_metrics.items():
--> 268 objective_value = objective.get_value(metrics)
269 # Support multi-objective.
270 if objective.name not in metrics:
~/comp_4447/dst2_env/lib/python3.9/site-packages/keras_tuner/engine/objective.py in get_value(self, logs)
53 The objective value.
54 """
---> 55 return logs[self.name]
56
57 def better_than(self, a, b):
KeyError: 'root_mean_squared_error'

Answer 1

What if :
objective=Objective("rmse", direction="min")
I don't know why it's not working. I tried it just now, modifying a notebook of mine to test, and 'rmse' works.
(p.s. I prefer Optuna though, for huge datasets, much faster)
Here are some snippets:
self.compile_metrics = ['AUC', 'binary_accuracy', 'Precision', 'Recall',
tf.keras.metrics.RootMeanSquaredError(
name='rmse', dtype=None
)]
self.model.compile(
optimizer = opt,
#loss = 'binary_crossentropy',
loss = 'mean_squared_error',
metrics = self.compile_metrics,
steps_per_execution = self.steps_per_execution,
)
# above are snippets from my_keras_tuner_class
dl = my_keras_tuner_class()
tuner = kt.BayesianOptimization(
dl.model_builder, # the hypermodel, found in custom class keras_tuning
#objective = 'val_binary_accuracy', # objective to optimize
objective = kt.Objective("rmse", direction="min"),
max_trials = nn_trials, # number of trials
distribution_strategy = tpu_strategy,
# this project name (a dir) is created in 'directory', and is used to
# resume tuning if the search is run again
directory = f'{workdir}',
project_name = 'kbayesian',
# if True, overwrite above directory if search is run again - i.e. don't resume
overwrite = True
Trial 2 Complete [00h 00m 16s]
rmse: 0.42514675855636597
Best rmse So Far: 0.42514675855636597
display(model.metrics_names)
-output:
['loss', 'auc', 'binary_accuracy', 'precision', 'recall', 'rmse']
# example trial
9s 453ms/step - loss: 2.5937 - auc: 0.6732 - binary_accuracy: 0.6353 - precision: 0.6174 - recall: 0.6237 - rmse: 0.5435 - val_loss: 1.1624 - val_auc: 0.7197 - val_binary_accuracy: 0.6970 - val_precision: 0.7083 - val_recall: 0.8500 - val_rmse: 0.5045