TensorFlow.js: Saving different model instances during training - neural-network

Question

I'm runing TensorFlow.JS on NODE and I'd like to be able to save a model during the training process at a certain point.
I tryed to just copy the actual model to a global variable but the JavaScript object is copied by reference and at the end the global variable has the same model than the last training epoch.
I then used many different JavaScript methods to do a deep clone (including lodash deep clone), but I get errors on the copied model like functions that end up missing (like model.evaluate).
I wonder if the only way I can save a certain checkpoint is directly using model.save() or if there is any other way to just copy (by value not reference) the model object to a global or a class property.
Thanks in advane!
** UPDATE **
Right now the best solution that has worked for me is by creating a copy of the model:
const copyModel = (model) => {
const copy = tf.sequential();
model.layers.forEach(layer => {
copy.add(layer);
});
copy.compile({ loss: model.loss, optimizer: model.optimizer });
return copy;
}
Consider that you may need to replicate some other settings from the original model to the new one (the copy).

Answer 1

A tf.Model object contains weight values, which usually live on the GPU
(as WebGL textures) and are not easily clonable. So it's not a good idea to
clone a tf.Model object. You should serialize it and save it somewhere.
There are two options:
If you are in Node.js, you should have relatively ample storage space. Just
use Model.save() to "snapshot" models onto the disk, which can be loaded back
later.
If you prefer to avoid going through the filesystem, you can do the serialization and deserialization in memory. Using methods tf.io.withSaveHandler and tf.io.fromMemory(). See the example below:
const tf = require('#tensorflow/tfjs');
require('#tensorflow/tfjs-node');
(async function main() {
const model = tf.sequential();
model.add(tf.layers.dense({units: 1, inputShape: [3], useBias: false}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.randomUniform([4, 3]);
const ys = tf.randomUniform([4, 1]);
const artifactsArray = [];
// First save, before training.
await model.save(tf.io.withSaveHandler(artifacts => {
artifactsArray.push(artifacts);
}));
// First load.
const model2 = await tf.loadModel(tf.io.fromMemory(
artifactsArray[0].modelTopology, artifactsArray[0].weightSpecs,
artifactsArray[0].weightData));
// Do some training.
await model.fit(xs, ys, {epochs: 5});
// Second save, before training.
await model.save(tf.io.withSaveHandler(artifacts => {
artifactsArray.push(artifacts);
}));
// Second load.
const model3 = await tf.loadModel(tf.io.fromMemory(
artifactsArray[1].modelTopology, artifactsArray[1].weightSpecs,
artifactsArray[1].weightData));
// The two loaded models should have different weight values.
model2.getWeights()[0].print();
model3.getWeights()[0].print();
})();